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EVALUATION 

* 


The  need  for  developing;  new  tools  and  techniques  for  producing  mor  e 
reliable  low  cost  software,  as  noted  in  such  c©etrmd(js  -fch*  CortwBuic, 
Control  Information  Processing  CCIP-85  Study  (Information  Processing/Data 
Automation  Implications  of  Air  Force  Command  and  Control  Requirements  in 
the  1980’s),  has  led  to  attempts  to  analyze  the  nature  and  types  of  soft- 
ware errors  in  order  to  be  able  to  accurately  predict  error  occurences, 
and  to  be  able  to  accurately  predict  the  reliability  of  software  produced. 
Many  different  types  of  models  of  the  software  debugging  process  have 
been  formulated  for  inis  purpose.  However,  a technique  that  has  not  been 
adequately  applied  to  software  errors  is  the  use  of  statistical  regression 
analysis  to  relate  error  occurrences  to  various  structural,  complexity, 
and  programmer-related  characteristics  of  a software  package. 

This  effort  was  initiated  in  response  to  the  CCIP-85  Studv  and  this 
need  for  applying  regression  techniques  to  software  errors,  and  fits  into 
the  goals  of  RADC  TPO  No.  5,  Software  Cos'  Reduction  (formerly  FADC  TPO 
No.  11,  Software  Sciences  Technology),  in  particular  the  area  of  Software 
Quality  (Software  Modeling).  The  report  focuses  on  the  analysis,  using 

multiple  linear  regression  techniques,  of  software  error  data  and  related 

£ * 

structural,  complexity,  and  programmer-related  variables  extracted  from 
two  large  Department  of  Defense  command  and  control  software  projects 
totalling  over  250,000  lines  of  higher  order  language  source  code.  The 
importance  of  performing  this  analysis  is  that  it  represents  the  first 
attempts  to  use  linear  regression  techniques  for  comparing  different 


f?  r trt ' 


‘ .M 


"4  “hr 


ax 


software  projects,  in  order  to  determine  those  characteristics  that  statis- 
tically impact  on  the  occurrence  of  software  errors. 

The  conclusions  drawn,  under  this  ..analysis  will,  therefore,  provide 
new  insights  into  those  factors  that  influence  software  errors.  In 
addition,  results  of  this  analysis  will  be  used  as  the  basis  for  future, 
continuing  analysis  for  collection  errors  using  regression  techniques,  and 
will  provide  a baseline  for  collection  of  software  characteristics  on 
future  software  projects  that  will  support  regression  analysis  of  soft- 
ware errors.  Finally,  the  results  of  this  and  future  similar  statistical 
analysis  efforts  will  provide  the  necessary  understanding  of,  and  insights 
into,  the  software  development  process,  through  the  understanding  of  those 
factors  that  lead  to  the  occurrence  of  software  errors,  chat  is  required 
in  order  to  produce  the  high  quality,  low  cost  software  desired. 

— 

ALAN  N.  SUKEKT,  Captain,  USAF 
Project  Engineer 
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1 . 0 INTRODUCTION 


This  document  is  the  final  technical  report  for  the 
Statistical  Prediction  Model  study,  RADC  Contract  No.  F30602- 
76-C-0213.  This  eight  month  study  focused  on  the  statistical 
prediction  of  programming  errors  using  a wide  range  of  pro- 
gram structure/complexity  variables  and  selected  programmer 
variables  as  predictors.  The  authors  view  this  study  as  one 
of  many  continuing  studies  which  need  to  be  performed  that 
further  investigate  how  programmers,  program  characteristics, 
management  methods,  and  software  testing  and  design  factors 
influence  and  contribute  to  errors  in  programs. 

This  report  is  organized  topically  into  eight  interrelated 
sections  and  one  technical  appendix.  A description  of  the 
major  topics  covered  in  each  section  is  as  follows: 

e Section  1.0  presents  a background  discussion  on  the 
role  and  importance  of  the  prediction  of  programming 
errors  for  improving  software  quality  and  reliability. 

The  purpose  of  this  study  is  also  stated  in  this  section. 

• Section  2.0  presents  a detailed  discussion  of  background 
information  on  the  two  data  samples  that  were  analyzed 
by  this  study.  This  discussion  focuses  on  the  software 
development  environment,  software  testing  considerations, 
programming  error  definition  and  classification,  predictor 
variables,  and  statistical  characteristics  of  both  samples. 

• Section  3.0  briefly  discusses  selected  limitations  of 

the  data  in  both  samples  that  could  affect  the  predictability 
of  errors  in  this  study.  Also,  three  important  preliminary 
analysis  findings  are  discussed  as  they  relate  to  the 
error  prediction  equations  that  were  developed  for  this 
study. 
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• Section  4.0  presents  a brief  discussion  of  the  multiple 
linear  regression  analysis  method  and  the  stepwise 
regression  procedure  that  were  used  to  obtain  the  error 
predictions  for  this  study. 

• Section  5.0  discusses  various  operational  and  analytical 
considerations  and  decisions  made  which  pertain  to  the 
error  prediction  equations  developed  for  this  analysis. 

• Section  6.0  presents  a summary  of  the  technical  results 
of  this  study  as  they  pertain  to  the  predictability  of 
errors  and  error  rate  in  both  data  samples.  A detailed 
presentation  and  discussion  of  the  results  for  each 
sample  follows  this  summary. 

• Section  7.0  presents  additional  analysis  of  the  error 
rate  per  program  measure  and  its  relationship  with  the 
programmer  rating  and  workload  variables,  and  also  its 
interesting  relationship  with  total  source  instructions 
as  observed  in  this  study. 

• Section  8.0  contains  the  major  conclusions  and  recom- 
mendations of  this  study.  Recommendations  are  discussed 
which  pertain  to  (1)  improving  the  consistency  of  pre- 
diction of  programming  errors  in  future  software  error 
prediction  studies,  (2)  further  research  requirements, 
and  (3)  proposed  data  collection  support  tools  and 
techniques. 

• Appendix  A contains  a brief  technical  discussion  of  the 
measure  of  programming  errors  to  be  analyzed  in  software 
error  prediction  studies  and  its  dependence  upon  the 
method  of  testing  used  during  the  software  test  period. 
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1.1  Background 


Software  quality  has  many  facets,  Including  availability, 
ease  of  maintenance,  CPU  run-time  required,  and  reliability. 
Precise  definitions  of  quality  or  its  components  do  not  exist. 
However,  software  reliability,  in  spite  of  a lack  of  quantita- 
tive definition,  has  received  a great  deal  of  attention  and 
rightly  so,  due  to  its  pervading  influence  on  the  other  aspects 
of  quality.  Without  a reliable  software  system,  availibility , 
run-time,  maintenance,  etc.,  are  meaningless. 

In  spite  of  the  ambiguity  and  lack  of  consistency  in  the 
definitions  of  software  reliability,  one  thing  common  to  them 
all  is  that  they  do  include  programming  errors.  The  analysis 
and  prediction  of  programming  errors  then  becomes  an  area  that:, 
would  contribute  to  the  assessment  and  improvement  of  software 
quality. 

Analysis  of  errors  becomes  important  from  the  standpoint 
of  determining  their  possible  causes  so  that  management  con- 
trols may  be  exercised  to  reduce  them.  Prediction  is  important 
as  a tool  for  the  analysis,  as  well  as  contributing  to  the 
testing  and  certification,  of  a software  system.  If,  for 
example,  fewer  errors  have  been  found  than  a prediction  formula 
indicates  are  present,  then  more  testing  should  be  performed. 

One  approach  that  is  being  considered  for  providing 
researchers  and  management  with  a more  definitive  understand- 
ing of  factors  which  affect  software  quality  and  reliability 
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is  that  of  predicting  programming  errors  through  the  use  of 
statistical  regression  models.  It  is  through  the  use  of  these 
prediction  models  that: 

(1)  estimates  of  the  software  reliability  during  each 
phase  of  the  software  development  effort  could  be 
provided; 

(2)  the  amount  of  further  testing  required  to  achieve 
or  insure  a given  level  of  program  quality  could 
be  estimated;  and 

(3)  the  relative  effects  of  selected  management  control  and 
both  design  and  coding  techniques  on  the  reduction 

of  programming  errors,  as  aids  for  improving  soft- 
ware quality  and  reliability,  could  be  assessed. 

Although  recent  studies  have  demonstrated  the  feasibility 
of  using  linear  models  and  regression  analysis  methods  to  pre- 
dict errors  in  programs,  additional  studies  are  needed  which 
apply  these  methodologies  in  order  to  assess  their  value  and 
importance  for  error  prediction  purposes. 

1 , 2 Purpose 

The  purpose  of  this  study  is  to  apply  the  statistical 
method  of  multiple  linear  regression  analysis  to  predict  pro- 
gramming errors,  using  a variety  of  variables  which  relate 
to  programmer  capability,  program  structure,  and  program 
complexity  as  predictors  of  errors.  Two  distinct  data  samples 
will  be  analyzed  by  this  study,  bach  sample  contains  data 
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on  programming  errors  and  program  characteristics  collected 
for  purposes  other  than  this  study,  and  provided  by  RADC. 
Both  samples  reflect  software  that  was  developed  by  inde- 
pendent contractors  for  different,  large-scale,  command  and 
control  applications.  A combined  total  of  5539  programming 
errors,  resulting  from  783  programs  with  a total  of  296,595 
source  instructions,  are  analyzed. 


2.0  DESCRIPTION  AND  CHARACTERISTICS  OF  DATA  SAMPLES 

Two  distinct  samples  of  data  were  analyzed  during  this 
study.  Each  data  sample  contains  data  on  both  programming 
errors  and  numerous  program  characteristics.  Throughout  this 
report  the  program  characteristics  that  are  presented,  dis- 
cussed, and  analyzed  are  referred  to  as  program  structure/program 
complexity  variables.  Although  much  of  this  program  charac- 
teristic data  represents,  for  example,  counts  of  the  number  of 
program  instructions  of  a certain  type  that  may  have  appeared 
i . the  program,  each  individual  program  characteristic  vari- 
able can  be  assumed  to  be  one  of  a variety  of  measures  or 
estimates  of  the  program  structure  and/or  complexity. 

The  two  data  samples  are  referred  to  as  sample  S and 
sample  T,  respectively,  throughout  this  report.  The  following 
sections  of  this  report  present  background  information  and 
other  relevant  statistics  concerning  the  software  development 
environments,  testing  considerations,  and  error  data  charac- 
teristics of  each  sample  which  will  be  of  importance 
when  evaluating  the  prediction  equation  results  obtained  from 
each  of  the  samples.  Table  2-1  presents  a brief  summary  of 
the  differences  which  existed  between  each  of  these  samples  on 
various  gross  level  project  and  program  characteristics. 
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TABLE  2-1.  CHARACTERISTICS  OF  DATA  SAMPLES 


^ J 
4-  m kfi  Ps 
CM  fA  TB  4-  » 

•>  *t  r* 
tA  CN 


< < 
• Na  r 
to  Z O 0l 
J OUJ 
to  ffi  — I-  J 

U > h-  z < 

- < « z 

H tt  Q O 

£ to  LU  ~ s w 

<-Z_ltU(- 
Ot  Ot  £ < U «£ 
O LU  < > z 2 

Ohtt  < UJ 
Dt  U OU  h 0. 

a.  < o z a.  o 

tJKa«iu 

SLttUQ 
3 U Z 
HUUQ<< 


to 

< — to  o >■ 

<4 

H 

at  ct  iiiN  j 

CO 

u 

O UJ  h-  h*  z 

J- 

FA  O' 

WA 

Q Z 

UJ 

Oh-  < o 

LU  rA 

if  FA  FA 

O' 

Z < 

3 

ct  u (j  2 

-J  LA 

CN  LA  FA 

t 

< at 

o 

Q.  < ON  z U>  UJ 

Q. 

s s 

£ *“ 

W QC  >•  — LU  to 

5 

•M  FA 

2 z 

a. 

< -J  O'  1-  < 

00 

O UJ 

00IZ3ZI 

</> 

u a 

fA 

K1UOQ-1 

« o 

I 

(O 

(-  at 

►— 

UJ 

o a: 

O 

UJ 

-J 

3 UJ 

Z it 

U9 

to 

a 

3 

at 

UJ 

UJ 

< 

z 

o 

C 

h-  O 

-J 

h- 

3 

o 

i- 

o 

to  Z 

< 

< o 

u 

£ 

z — 

£ 

at 

uj  z 

H 

- £ 

5 

at  < 

u 

a 

£ 

£ 

at 

at 

< -J 

z 

Ul 

< 

UJ  < 

o 

o 

o a 

at 

a 

a ct 

o 

at 

Z <J 

U_  UJ 

a to 

o 

at  O 

at 

at 

o z 

£ 

Ul 

o 

3 O 

Q. 

UJ 

hM  M 

U £ 

U _J 

at 

O £* 

1-  £ 

O 5 

O 03 

a. 

to  a. 

UJ 

UJ 

< £ 

at 

< 

(J 

u 

o < 

o 

at  <J 

e*  —■ 

-i 

-j  -i 

< 

< 

- at 

UJ 

UJ  O 

uj  at 

< < < 

at 

ot 

-1  o 

to 

a at 

eo  < 

h- 

h-  1- 

UJ 

UJ 

a.  o 

3 £ 0.  £ > 

O 

O O 

> 

> 

Q-  at 

3 

3 

t- 

h-  h-  < 

< 

< a. 

Z 

Z 

/ V r 


; - • •'v  o. 


TOTAL  SOURCE  INSTRUCTIONS 


2.1  Sample  S 


2.1.1  Software  Development  Environment 

Sample  S software,  which  consisted  of  534  programs,  was 
developed  as  three  command  and  control  systems  which  for  the 
purposes  of  this  report  are  to  be  referred  to  as  projects  M, 

B,  and  P,  respectively.  This  software  was  developed  jointly  by 
two  private  industry  organizations  from  mid-1969  through  late 
1973,  and  represents  an  effort  of  approximately  5500  man- 
months.  These  programs  consisted  of  181,249  source  instruc- 
tions written  in  CENTRA:! , an  English- like,  special  purpose, 
higher  level  language  that  was  designed  for  use  only  on  Central 
Logic  and  Control  (CLC)  computers.  The  programs  analyzed 
represent  about  80  percent  of  the  approximately  644  programs  with 
240,000  source  instructions  that  were  written  for  the  entire 
command  and  control  application.  The  software  development 
effort  attempted  to  conform  to  the  traditional  approach  of 
building  large  software  systems: 

(1)  definition  of  system  performance  requirements, 

(2)  design  of  functional  specifications  from  which  pro- 
gramming specifications  are  written, 

(3)  coding  and  unit  testing  of  those  software  elements 
comprising  a process  sub function , 

(4)  integration  testing  of  the  elements  within  a 
subfunction,  and 

(5)  integration  testing  of  the  system  processes. 
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Significant,  but  not  always  successful,  attempts  were  made 
to  write  system  design  specifications  and  develop  software  be- 
fore the  system  performance  requirements  and  functional  speci- 
fications were  completed.  In  projects  M and  P for  example, 
serious  program  design  work  and  coding  were  initiated  during  the 
period  of  June  through  August  1969,  although  the  performance 
requirements  were  not  completed  until  June  1970.  Using  the  per- 
formance requirements  as  a base,  a design  team  from  one  of  the 
organizations  defined  the  system  functional  design  requirements 
and  specifications  from  which  software  design  teams  from  both 
organizations  generated  the  formal  programming  design  specifi- 
cations. Where  these  latter  specifications  did  not  coincide 
with  the  software  already  written,  the  original  code  had  to  be 
rewritten. 

A description  of  the  extent  to  which  the  new  programming 
technologies  (e.g.,  structured  programming,  top-down  design, 
etc.)  were  implemented  and  applied  during  the  development  of 
each  of  the  three  sample  S software  projects  is  presented  in 
Table  2-2  along  with  several  basic  program  length  and  error 
statistics  for  each  project.  It  appears  from  Table  2-2  that 
project  P was  the  only  project  that  actually  implemented  or 
applied  these  new  technologies  to  any  large  extent. 

It  is  important  to  note  here  one  final  consideration  regard- 
ing these  new  technologies  as  applied  in  sample  S.  No  informa- 
tion is  available,  other  that  that  which  appears  in  Table  2-2, 
to  give  any  indication  of  how  consistently  and  with  what 
thoroughness  and  quality  the  concepts  and  principles  which 
underlie  these  new  technologies  were  strictly  followed  in  the 
programs  developed  for  sample  S.  Necessarily  then,  and  in  line 
with  the  stated  purposes  of  this  research  study,  no  comparisons 


TABLE  2-2.  APPLICATION  OF  NEW  PROGRAMMING  TECHNOLOGIES 

IN  SAMPLE  S PROJECTS 


M 

B 

P 

NO.  OF  PROGRAMS 

395 

104 

35 

TOP  DOWN  DESIGN 

1.0U3) 

100.0%C104) 

48 . 6% ( 1 7 ) 

STRUCTURED  CODE 

2 , 3 % C 1 1 ) 

0 . 0%  (0 ) 

65 .7%C23) 

CHIEF  PROGRAMMER 

0 . 0%(0) 

0 . 0% (0  ) 

5 1 . 4%  C 1 8 5 

TEAM 

PROGRAMMING 

0 . 0%C0) 

0.0K0) 

97. 1%(34) 

LIBRARIAN 

AVERAGE  SOURCE 

345 . 2 

212.3 

651.9 

INSTRUCTIONS 

AVERAGE  ERRORS/ 

6.8 

6.0 

6.8 

PROGRAM 

AVERAGE  ERROR  RATE 

2.0 

2.8 

1.0 

•average  ERROR  RATE  = A MEASURE  OF  TOTAL  ERRORS  FOUND  PER 
100  LINES  OF  SOURCE  CODE  FOR  EACH  PROJECT. 


are  made  or  inferences  drawn  concerning  the  relative  effective- 
ness of  these  technologies  for  improving  software  quality  and 
reliability. 

2.1,2  Software  Testing  Considerations 


Each  of  the  three  software  development  projects  of  sample 
S adhered  to  a basically  standard  code  writing  and  testing 
scheme.  One  programmer  was  usually  assigned  the  responsibility 
of  writing  several  programs  which  would  subsequently  interface 
as  a single  functional  unit,  with  several  of  these  units  forming 
a subfunction.  When  a program  was  compiled  error-free  and  unit 
tested,  it  was  combined  with  its  related  counterparts  to  form 
a functional  unit  for  element  testing.  This  testing  was  per- 
formed by  the  programming  team  and  was  usually  the  prerequisite 
for  submitting  a system's  subfunctions  to  the  Test  and  Integra- 
tion (TSI)  team.  Immediately  prior  to  the  TSI  phase,  all  the 
subfunctions  of  a system  would  be  bound  together  to  form  that 
system's  thread.  It  was  this  thread  which  was  delivered  to 
TSI  for  integration  testing.  The  TSI  phase  assured  that  the 
subfunctions  of  a system  interfaced  properly  and  represented 
the  beginning  of  formal  error  recording  for  the  software  being 
tested.  Trouble  reports  describing  the  error  and  its  severity 
were  forwarded  to  the  respective  programming  team  for  resolu- 
tion. To  expedite  error  resolution,  patches  to  the  object 
code  were  made  prior  to  resubmission  to  the  TSI  group.  The 
applicable  source  code  was  subsequently  updated,  usually  when 
a new  version  of  a process  was  released. 

The  system  integration  tests  were  designed  to  verify  that 
the  system  could  respond  in  certain  areas  of  basic  system 
capabilities  within  predicted  tolerances.  The  testing 
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required  to  support  the  final  demonstration  or  acceptance  tests 
in  each  test  case  was  assumed  to  begin  with  single-site  tests, 
in  which  communications  with  the  rest  of  the  system  were  usually 
simulated.  Later  tests  were  built  up  to  include  all  sites 
netted  together.  Table  2-3  presents  the  software  development 
and  testing  dates  that  were  reported  for  each  of  the  three 
projects  of  sample  S. 

No  other  specific  data  was  available  for  this  study  with 
respect  to  the  amount  and  thoroughness  of  testing  that  sample 
S programs  underwent  during  the  TSI  phase  of  software  development. 

2.1.3  Definition , Classification,  and  Collection  of  Error  Data 


For  sample  S,  programming  errors  were  defined  as  those 
errors  found  during  the  TSI  phase  which  could  be  attributed  to 
the  programmer  and  required  a change  to  the  program's  source 
coda.  At  the  time  this  error  data  was  provided,  no  additional 
classification  of  errors  had  been  attempted.  (Presently, 
however,  there  is  an  effort  underway , supported  by  RADC,  that 
will  result  in  a thorough  classification  of  the  various  error 
types  for  the  sample  S programs  being  analyzed  for  this  report.) 

With  respect  to  error  data  collection  for  sample  S programs, 
early  in  this  analysis  effort  ini  omul  d:i  scu:-r  ionr  with  per- 
sonnel responsible  for  the  sample  S data  collection  revealed 
that  the  data  collection  on  program  character istics  may  have 
been  obtained  as  much  as  three  years  after  the  error  data  was 
collected.  This  delay  is  estimated  based  on  the  dates  provided 
in  Table  2-3.  After  the  errors  were  detected,  programs  were 
modified  as  a direct  result  of  these  errors.  Programs  continued 
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PREDICTOR  VARIABLE  VALUES  FOR  SAMPLE  S)  WERE  COLLECTED,  AS  I 
FOR  THIS  STUDT.  THE  VERSION  FOR  PROJECT  M WAS  NOT  REPORTED 


to  be  modified  after  the  TSI  phase  as  part  of  ithe  normal 

growth  process  of  the  respective  systems  being;  developed. 

i 

Since  each  of  the  predictor  variables  in  sample  S represent 
program  characteristic  variables,  and  given  that  there  was  no 
measure  cf  the  extent  to  which  these  program  characteris- 
tics had  been  modified  between  the  time  when  the  error 
and  predictor  variable  data  were  collected,  a serious  question 
arose  as  to  the  validity  of  the  results  obtained  from  a multiple 
regression  analysis  of  this  data.  This  condition  might  result 
in  the  anomalous  position  of  attempting  to  predict  errors  from 
data  which  to  some  extent  at  least  may  have  resulted  from  the 
errors. 

Additional  discussion  with  personnel  responsible  for  the 
sample  S data  collection  indicated  that  the  extent  of  the 
modifications  made  to  the  program  characteristics  as  a result 
of  the  errors  being  corrected  was  minor.  However#  it  was 
apparent  from  having  initially  reviewed  the  sample  S data 
that  the  variability  of  some  of  the  predictor  variables  was 
also  minor.  Thus,  the  effect  of  the  minor  modifications  on  the 
conclusions  drawn  from  predictor  variables  with  minor  vari- 
ability became  an  unknown  which  could  not  be  assessed. 

In  order  to  deal  with  this  problem  in  such  a way  that  the 
data  could  still  be  utilized  for  error  prediction  purposes, 
a decision  was  made  to  categorize  each  of  the  independent 
variables  for  this  sample.  This  categorization  procedure 
involved  examining  the  range  of  each  independent  variable, 
grouping  the  scores  into  equal  intervals,  and  the  assigning 
of  new  scores  to  the  intervals.  For  example,  for  one  variable 
all  scores  in  the  interval  0-99  would  be  rescored  as  1;  scores 


2-9 


f -n  **  ^ ' 


/*>■ 


*«* 


in  the  interval  from  100-199  would  be  rescored  as  2,  etc.. 
Clearly,  this  grouping  technique  when  implemented  would  un- 
doubtedly throw  away  some  useful  data;  it  would  also  eliminate 
part  of  the  error  due  to  assuming  that  code  changes  after 
error  counts  were  insignificant.  If  modifications  to  the 
program  characteristics  were  indeed  minor  as  had  been  indicated, 
then  categorizing  the  data  would  give  a truer  picture  of  what 
these  characteristics  would  have  been  at  error  collection  time 
for  all  values  of  the  predictor  variables  except  those  at  the 
boundaries  of  the  class  intervals.  Thus,  conclusions  drawn 
would  have  relatively  more  validity. 

The  categorization  procedure  was  applied  to  all  sample  S 

predictor  variable^.  Intercorrelation  matrices  were  then 

/ 

obtained  for  each  of  the  projects  which  showed  the  inter- 
correlations among  the  predictors  and  between  the  predictors 
and  the  criterion  variable,  programming  errors.  Each  of  these 
sets  of  intercorrelations  was  then  compared  with  the  respective 
intercorrelations  obtained  when  the  original  (uncategorized) 
predictor  variable  values  were  used.  Essentially,  no 
significant  statistical  differences  were  found  among  the  two 
sets  of  correlation  matrices  for  each  of  the  projects.  Thus, 
after  having  performed  this  rather  extensive  computational 
task  of  categorizing  the  data  and  then  comparing  various  sets 
of  correlation  matrices,  a firm  decision  was  made  to  continue 
on  with  the  multiple  linear  regression  analysis  using  the 
original  data  as  obtained  for  sample  S. 

Based  on  the  above  discussi.on  of  problems  associated  with 
the  sample  S error  data  collection,  it  should  be  noted  tnat 
any  data  that  is  to  undergo  a secondary  analysis  is  subject 
to  the  same  or  similar  problems  that  were  discovered  when 
attempting  to  analyze  this  data.  Accessibility  of  the  sample  S 
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personnel  led  directly  to  the  discovery  of  these  problems. 

Other  data  in  the  future  could  be  collected  by  personnel  not 
as  accessible,  thus  resulting  in  problems  at  least  as  serious, 
though  undiscoverable. 

The  problems  discovered  thus  far  lead  to  a strong  argument 
for  (1)  identifying  or  determining  what  use  is  to  be  made  of 
the  data  before  it  is  collected,  and  (2)  developing  and 
providing  the  necessary  data  collection  instruments,  procedures, 
formats,  and  software  support  systems  that  can  collect  and 
store  repeated  snapshots  of  program  characteristics  and  error 
data  throughout  various  phases  of  the  software  development 
cycle . 

2.1.4  Predictor  Variables 


A list  and  description  of  the  54  predictor  variables 
that  were  made  available  for  the  analysis  of  sample  S programs 
is  presented  in  Table  B-l  of  Appendix  B.  Variables  56  through 
109  were  constructed  during  this  study  in  order  to  investigate 
their  effectiveness  in  predicting  error  rate/program  for  this 
sample.  Further  discussion  of  these  variables  (i.e.,  56-109) 
is  presented  in  section  3.0. 

Each  of  these  initial  54  predictors  was  collected  as 
previously  mentioned  during  December  1974  and  January  1975 
via  an  automatic  scanner  program  developed  by  sample  S data 
collection  personnel.  This  scanner  program  could  interrogate 
source  code  programs  that  were  written  in  CENTRAN,  ALC,  or 
PL/1. 


It  is  important  to  note  that  other  variables  in  addition 
to  the  54  that  were  collected  could  have  been  measured  and 
collected  via  this  scanner  program.  For  example,  Lock  Macros 
(variable  X20)  is  the  only  one  of  several  different  types  of 
CENTRAN  system  macros  that  was  collected  separately.  All  other 
system  macros  used  in  the  program,  including  hock  Macros,  are 
summed  into  variable  X10,  System  Macros.  Basically  then, 
what  variables  were  collected  by  sample  S personnel  clearly 
represent  the  variables  hypothesized  by  th&t  group  to  be  of 
some  particular  interest  and  importance  for  their  own  purposes 
of  analysis.  The  point  made  here  is  that  other  variables  which 
may  have  contributed  significantly  to  the  prediction  of  pro- 
gramming errors  were  not  collect-d  in  the  sample  S data  and 
resultantly  were  not  available  for  review,  analysis,  and 
evaluation  by  this  study.  There  is  a definite  need  for  the 
definition  of  a uniform  set  of  program  characteristics  that 
may  be  applied  to  a wide  variety  of  projects.  By  doing  so, 
it  would  be  possible  to  compare  the  results  of  one  project  with 
another  and  thus  draw  conclusions  applicable  to  programming  in 
general,  not  merely  to  programming  as  reflected  by  one  specific 
project. 

Lastly,  with  respect  to  the  first  54  predictors  it  is 
important  for  analysis  purposes  to  identify  the  various  linear 
combinations  that  existed  among  these  variables.  Variable  XI 
was  identified  as  a linear  combination  of  variables  X7,  X8,  X10, 
Xll , X15,  X16,  X41,  and  a variable  that  was  not  separately 
counted,  the  number  of  assignment  instructions  not  involving 
arithmetic  operations  (e.g.,  A*l;  B«5  etc.).  System  Macros  (X10) 
is  a linear  combination  of  variables  X3,  X9,  X20,  and  any  other 
CENTRAN  System  Macros,  Variable  X17,  Scaling/Rounding  Operations, 


is  considered  a part  of  Centran  Functions  (X16) . Variable 
X39,  Total  Variables,  is  a linear  combination  of  variables  X22, 
X25,  X27,  X29 , X31,  X33,  X35,  and  X37.  Variable  X40,  Total 
Variable  Frequency,  is  a combination  of  variables  X23,  X24,  X26, 
X28,  X30,  X32,  X34,  X36,  and  X38.  Finally,  variable  X41, 

Total  Do  Loops,  is  a combination  of  variables  X18,  X19,  X42, 

X43,  X44 , X45 , X46,  and  X47. 

2.1.5  Characteristics  of  Sample  Data 

During  the  initial  phases  of  the  analysis,  univariate  and 
bivariate  frequency  distributions  ware  obtained  for  each  of 
the  54  predictors  and  for  each  of  the  predictor  variables  with 
errors/program.  For  the  dependent  variable  and  most  of 
the  predictor  variables  the  univariate  frequency  distributions 
were  asymetric  with  the  higher  frequencies  concentrated 
toward  the  lower  end  of  the  variable  and  the  smaller  frequencies 
asymptotically  spread  out  toward  the  higher  end.  The  bivariate 
distributions  basically  showed  the  existence  of  a predominant 
number  of  very  low  to  moderate  linear  relationships  existing 
between  the  various  predictor  variables  and  programming  errors. 
There  was  a clear  indication  of  a tendency  toward  non-linearity 
in  many  of  the  relationships  between  the  predictor  variables 
and  errors.  Based  on  these  observations  it  is  possible  that 
better  predictions  could  be  obtained  by  non-linear  transforma- 
tions, of  either  or  both  programming  errors  and  the  predictor 
variables . 

in  addition,  based  on  the  sample  S project  statistics 
as  presented  in  Table  2-4  and  on  differences  that  were  observed 
between  means  and  measures  of  variability  of  predictors  be- 
tween projects,  there  was  a clear  indication  that  the  three 


TABLE  2.4  SAMPLE  S PROJECT  STATISTICS 


STATISTICS 

M 

B 

P 

ALL 

PROJECTS 

NO.  OF  PROGRAMS  CWITH  S 

395 

104 

35 

534 

WITHOUT  ZERO  ERRORS,) 
TOTAL  SOURCE  INSTRS . 

136,353 

22,075 

22,816 

181,249 

TOTAL  ERRORS 

2,673 

622 

238 

353  3 

AVG.  PROGRAM  LENGTH 

345.21 

212.26 

651.89 

339.42 

AVG.  ERRORS/PROGRAM 

6.77 

5.98 

6.80 

6.62 

AVG.  ERROR  RATE 

1.96 

2.82 

1.04 

1.95 

AVG.  ERROR  RATE/PGM. 

2.42 

3.26 

1.29 

2.51 

PERCENT  OF  CODE  WITH 
ZERO  ERRORS 

9.m 

5.40% 

0.00% 

3,04% 

PERCENT  OF  PROGRAMS 
WITH  ZERO  ERRORS 

22.78% 

8.65% 

0.00% 

18.54% 

NO.  OF  PROGRAMS  CWITH 

305 

95 

35 

435 

1 OR  MORE  ERRORS) 

TOTAL  SOURCE  INSTRS. 

122,971 

20,882 

22,816 

166,669 

TOTAL  ERRORS 

2,673 

622 

238 

3533 

AVG.  PROGRAM  LENGTH 

403.17 

219.81 

651.89 

424.96 

AVG.  ERRORS/PGM. 

8.76 

6.55 

6.80 

8.12 

AVG.  ERROR  RATE 

2.17 

2.9? 

1.04 

2.06 

AVG.  ERROR  RATE/PGM. 

3.13 

3.57 

1.29 

3.08 

NO.  OF  PROGRAMS  (WITH 

90 

9 

0 

99 

ZERO  ERRORS) 

TOTAL  SOURCE  INSTRS. 

13,387 

1193 

14,580 

AVG.  PROGRAM  LENGTH 

i 148.74 

132.56 

— 

140.65 

| SHORTEST/LONGEST  PGMS . 

, 6/1047 

18/389 

- “ - 

6/1047 
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sample  S projects  were  quite  heterogeneous  over  a majority  of 
the  data  variables.  As  indicated  in  Table  2-4  many  of  the 
programs,  both  short  and  long,  were  reported  as  error  free 
as  a result  of  testing  during  the  Til  phase.  Particularly 
for  the  longer  programs,  this  fact  is  hard  to  reconcile  with 
past  experience  and  logical  reasonong  on  this  subject  which 
would  contend  that  as  programs  become  longer,  the  number  of 
paths  through  the  program  increases,  which  in  turn  increases 
the  program '8  complexity,  thus  increasing  the  likelihood  of  a 
larger  number  of  errors  residing  in  these  longer  programs. 
Section  3.0  comtains  additional  discussion  about  programs  with 
zero  reported  errors. 
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2 . 2 Sample  T 


2.2.1  Software  Development  Environment 


Sample  T software  consists  of  249  programs  which  repre- 
sent the  initial  delivery  of  a large  command  and  control 
software  system.  This  system  was  functionally  developed  as 
eight  subsystems  which  for  the  purposes  of  this  report  are 
referred  to  as  subsystems  A through  H,  respectively.  The 
software  was  jointly  developed  by  two  private  industry  organi- 
zations, with  one  of  the  organizations  being  responsible  for 
77  of  the  249  programs  that  were  written.  The  total  programs 
consisted  of  115,346  source  instructions  written  in  JOVIAL  J4, 
a higher  level  language  which  is  compatible  with  the  SYMON 
operating  system.  Batch  type  processing  was  the  exclusive 
operating  mode  used  during  this  software  development. 

Software  development  for  sample  T subsystems  followed  a 
"single  increment"  development  approach  and  was  goverened  by 
formally  specified  and  approved  requirements  that  had  been 
defined  down  to  the  function  level.  :Single  increment",  as 
defined  for  sample  T data,  refers  to  a typical  development  cycle 
in  which  each  development  phase  is  performed  only  once.  This 
is  in  contrast  to  the  top-down,  multiple  increment  approach, 
where  the  cycle  is  repeated  several  times,  first  for  a system 
of  stubs  and  subsequently  when  the  stubs  are  replaced  with 
deliverable  (i.e.,  ready  to  be  tested)  software. 

Structurally,  the  smallest  compilable  unit  of  source  code 
was  the  routine  or  program.  Programs  were  joined  to  form 
functions,  functions  were  joined  to  form  subsystems,  and 
finally  subsystems  were  joined  to  comprise  the  one  command 
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and  control  system.  This  structure  was  produced  by  a project 
organization  based  on  the  function  being  developed.  This 
meant  that  management  of  sample  T software  development  was 
set  up  In  conjunction  with  the  software  structure  so  that 
personnel  assigned  to  one  work  unit,  ranging  in  size  from  5 
to  15  programmers,  produced  all  the  software  in  one  or  more 
of  the  functions.  No  new  or  advanced  programming  technologies 
were  reported  as  being  applied  to  sample  T programs  or  imple- 
mented during  their  development. 

2.2.2  Software  Testing  Considerations 

Testing  of  sample  T software  proceeded  through  five 
sequential  phases j development,  validation,  acceptance,  inte- 
gration, and  operational  demonstration.  Development  testing 
was  informal  with  all  test  cases  being  written  and  executed 
by  the  program  development  personnel.  No  information  was 
provided  regarding  the  length  of  time  involved  in  this  partic- 
ular phase  of  testing.  Validation  testing  marked  the  first 
phase  of  formal  software  testing  by  an  independent  test  group. 
The  objective  of  tests  that  were  developed  for  both  the  valida- 
tion and  acceptance  phases  was  to  demonstrate  that  the  sample  T 
programs  did  in  fact  satisfy  or  meet  the  formal  functional 
requirements  that  had  been  specified.  Each  of  the  tests  applied 
during  these  two  phases  was  run  at  the  subsystem  level  but 
was  designed  to  examine  software  performance  at  the  program, 
function,  subsystem,  and  Bystem  levels.  An  important  considera- 
tion of  these  two  phases  was  that  all  testing  was  performed 
on  a master  configuration  and  no  alteration  of  the  code  was 
allowed.  Acceptance  testing  consisted  of  rerunning  selected 
tests  that  had  been  used  for  validation  testing,  particularly 
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those  that  specifically  demonstrated  the  software  requirements. 
Customer  acceptance  of  the  software  was  dependent  upon  the 
successful  completion  of  these  tests. 

Integration  testing  was  conducted  by  an  independent  con- 
tractor whose  major  responsibility  was  to  demonstrate  that 
the  applications  software  interfaced  correctly  with  the  operat- 
ing system  and  the  system  support  software.  Tests  conducted 
during  this  time  were  similar  in  structure  and  formality  to 
those  tests  that  were  used  during  the  validation  and  acceptance 
phases.  Tho  operational  demonstration  phase  was  a short  period 
of  testing  which  followed  an  operational  timeline  and  which 
used  an  operational  data  base.  The  objective  of  tenting  during 
this  period  was  to  demonstrate  the  satisfactory  porforr.ance 
of  the  software  in  the  operational  environment. 

The  approximate  length  of  calendar  time  (as  measured  in 
weeks)  involved  in  each  of  the  four  phases  of  formal  testing 
for  sample  T programs  was  as  follows: 


Validation 

10.3 

Acceptance 

1.5 

Integration 

9.5 

Operational  Demonstration 

2.3 

With  respect  to  error  detection  during  each  of  these  test 
phases,  it  was  reported  that  each  phase  found  errors  which 
should  have  been  detected  in  earlier  phases,  ho  measures  of 
the  amount  or  thoroughness  of  testing  were  available  for 
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•ample  T software.  Resultantly,  It  is  not  possible  to  state 
what  percent  of/  and  to  what  extent/  the  code  had  been  exercised 
(given  the  possible  range  of  input  values  for  each  test)  during 
this  formal  testing  period. 

2.2.3  Definition,  Classification,  and  Collection  of  Error  Data 


For  sample  7,  programming  errors  were  defined  as  those 
errors  found  during  the  four  stages  of  formal  testing  which 
could  be  attributed  to  the  programmer  and  required  a change 
to  the  program's  source  code.  Programming  errors  were  classi- 
fied as  belonging  to  one  of  a variety  of  error  categories  which 
either  (1)  described  the  symptoms  of  tho  problem  (e.g.,  output 
report  has  data  entries  that  were  not  printed  or  are  missing) 
or  (2)  actually  identified  the  cause  of  the  error  (e.g.,  logical 
condition  not  tested  which  resulted  in  abnormal  program  end) . 

The  2006  programming  errors  that  are  analyzed  in  this 
report  were  aggregated  and  classified  according  to  nine  major 
error  categories.  These  categories  along  with  the  approximate 
percentage  of  errors  that  occurred  in  each  are  as  follows i 


Logic 

Data  Handling 

Interface 

Data  Input/Output 

Computational 

Other 

Data  Base 

Data  Definition 


26.0 

18.2 

17.0 

16.4 
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Unfortunately,  for  the  sample  T error  data  provided  for 
this  study,  there  was  no  way  of  knowing  which  errors  belonged 
to  each  of  the  error  categories  as  indicated.  Only  recently, 
after  a majority  of  the  analysis  had  been  completed,  was  it 
possible  to  obtain  a first  available  version  of  the  automated 
data  which  describes  in  what  programs  the  particular  error  type 
occurred,  the  phase  of  testing  in  which  the  error  was  de- 
tected, etc.  Necessarily  then,  no  analysis  of  programming 
errors  by  error  type  or  category  was  attempted  during  this 
study. 

Two  important  points  regarding  the  error  categories  as 
listed  deserve  brief  mention  at  this  time.  First,  the  indi- 
viduals who  assigned  or  dlassified  programming  errors  according 
to  these  software  error  categories  were  not  the  same  individuals 
who  had  initially  recorded  the  error  in  a Software  Problem 
Report  (SPR) . The  SPR  and  a closure  report  which  contained  an 
explanation  of  the  problem  and  the  corrections  required  to 
resolve  the  problem  were  usually  the  main  sources  of  infor- 
mation used  to  retrospectively  classify  the  errors. 

Secondly,  it  was  reported  by  the  individuals  who  classified 
the  errors  that  not  all  errors  were  necessarily  programming 
errors,  and  that  the  probable  source  of  these  errors  could  be 
traced  to  four  development  activities s 

(1)  requirements  specifications, 

(2)  design, 

(3)  coding,  and 

(4)  maintenance  (correction  of  other  errors) . 
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When  the  sample  T programming  errors  were  classified  according 
to  these  probable  sources:  (1)  errors  were  only  able  to  be 

classified  either  as  design  or  coding  errors  based  on  the 
available  information  provided  by  the  SPP.  and  closure  report, 
and  (2)  approximately  64£  of  the  errors  could  be  attributed 
to  design  and  36*  to  coding  as  probable  sources  for  these 
errors.  An  explanation  that  was  offered  for  'this  outcome  was 
that,  in  their  collection  of  supporting  data  to  explain  software 
error  histories,  poorly  stated  requirements  or  changing  inter- 
pretations of  requirements  were  offered  as  reasons  for  diffi- 
culty in  developing  various  programs  that  were  found  to  be 
error-prone . 

As  previously  mentioned,  error  data  for  sample  T was 
collected  throughout  each  of  the  four  stayes  of  formal  systems 
testing.  However,  no  information  is  available  as  to  the  dates 
for  the  data  collection  of  the  program  structure  data.  If 
the  program  structure  data  was  collected  pribr  to  the  error 
data  collection,  then  the  programming  errors  could  be  considered 
as  immediately  resulting  from  the  various  measured  and  un- 
measured program  characteristics.  On  the  other  hand,  if  the 
program  structure  data  collection  was  performed  after  modifi- 
cations were  made  to  these  program  structure  variables  as  a 
direct  result  of  correcting  for  errors  found  during  the  formal 
test  phases,  then  there  exists  a similar  situation  as  was 
discovered  for  the  sample  S programs.  This  problem,  for  both 
samples,  is  one  of  several  which  raises  questions  as  to  the 
validity  of  the  data  that  was  used  for  this  study.  It  is 
important  that  researchers  are  aw^re  of  sources  of  possible 
invalidity  in  data  collection  or  program  selection.  The 
effects  can  then  be  considered  in  terms  of  the  results  obtained 
and  the  conclusions  drawn  from  the  study. 
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2.2.4  Predictor  Variables 


A list  and  description  of  the  16  predictor  variables 
that  were  provided  for  the  analysis  of  sample  T programs  is 
presented  in  Table  B-2  of  Appendix  B.  Variables  18  through 
31  were  constructed  during  this  study  to  investigate  their 
effectiveness  in  predicting  error  rate/program  for  this 
sample.  These  variables  will  be  further  discussed  in 
Section  3.0. 

Each  of  the  16  original  predictor  variables  were 
collected  by  means  of  a scanner  progrnm  which  could  interro- 
gate source  code  programs  that  were  written  in  JOVIAL  J4. 

The  only  known  linear  combinations  that  existed  among  these 
16  predictors  v-era  as  follows:  (1)  Total  Source  (TS)  was  a 

linear  combination  of  variables  Non-Executable  Instructions 
(NEX)  and  Executable  Instructions  (EX) ; and  (2)  Total  Inter- 
faces (TI)  was  a linear  combination  of  the  two  interface  vari- 
ables, Application  Interfaces  (AP)  and  System  Interfaces  (SYS) . 

Regarding  a more  detailed  description  or  definition  of 
the  software-related  predictor  variables,  only  a limited 
aim  unt  of  more  specific  information  was  available.  This  infor- 
mation is  as  follows:  four  generic  types  of  executable  code 

were  arbitrarily  defined: 

I/O  - I/O  refers  to  JOVIAL  defined  and  SYSTEM 
defined  input  and  output  statements. 

JOVIAL  I/O  statements  include  FORMItl, 

FORMOUT,  DECODE  and  ENCODE.  SYSTEM  DISC 
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I/O  includes  ' SDAHA  and  its  various  entrances. 
Examples  of  SYSTEM  TAPE  I/O  are  'CWRITE, 

' WEOF  and  'REWIND. 

COMPUTATIONAL-  These  are  statements  expressing  equations 
containing  arithmetic  operators. 

Examples  AA  - BB*CC**2/DD  $ 

DATA  HANDLING-  These  statements  effect  a simple  data 

transfer  (equality)  from  one  variable  to 
another  and  are  distinguished  from  compu- 
tational statements. 

Exampless  XX-YY  $,  AA($BB+2,  DD|)  - 'PR  $. 

LOGICAL-  Logical  statements  establish  branches  in 

the  code  and  include  the  IF,  IFEITH,  ORIF, 

FOR  and  GOTO  SWITCH  statements. 

Also,  more  descriptive  information  relating  to  how  the 
Total  Branches  (BR)  and  Interface  (AP,SYS)  variables  were 
defined  was  provided.  Total  Branches  (BR)  was  described  as 
including  all  possible  logical  branches,  resulting  from  IF, 
IFEITH,  ORIF,  and  GOTO-SWITCH  name  statements.  The  BR  variable 
does  not  reflect  the  actual  number  of  logical  branches  the 
program  will  make  when  it  executes.  Program-to-program  and 
program-to-data  base  interface  descriptions  were  described 
as  being  available  from  system  utility  or  construct  programs. 

To  this  could  be  added  details  of  the  individual  interface 
(e.g.,  number  of  arguments  in  the  calling  sequence),  the  type 
of  interface  (applications,  system,  user,  data  base),  and  the 
format  of  the  information  passed. 
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The  two  predictor  variables,  Programmer  Rating  (RAT) 
and  Workload  (WKLD) , were  metrics  that  were  constructed 
to  evaluate  programmer  performance  with  respect  to  (1)  selected 
programmer-specific  criteria  and  (2)  programmer  assignment  or 
job-specific  criteria.  The  evaluation  was  amde  on  only  those 
that  had  been  exclusively  developed  by  one  of  the  two  private 
industry  organizations  who  shared  responsibility  for  the  overall 
software  development  effort.  The  evaluation  was  performed  by* 
the  programmer ' s line  management  after  the  project  was  completed. 
Table  2-5  presents  the  personnel  evaluation  parameters  that 
were  used  to  develop  the  programmer  rating  and  workload  measures 
relative  to  software  quality.  From  these  parameters,  the 
Programmer  Rating  variable  (RAT)  was  constructed  by  simply  sum- 
ming the  scores  obtained  on  each  of  the  knowledge,  intelligence, 
initiative,  and  responsibility  categories.  One  final  point  to 
consider  is  that  many  of  the  172  programs  for  which  RAT  and 
WKLD  measures  were  available  were  developed  jointly  by  two  or 
more  (up  to  15)  programmers.  For  these  programs,  the  RAT  and 
WKLD  measures  represent  the  averages  obtained  for  these  vari- 
ables over  each  of  the  individual  programmer's  scoreB. 

2.2.5  Characteristics  of  Sample  Data 

Subsystem  statistics  for  sample  T are  presented  in  Table 
2-6.  Clearly  these  subsystems  are  quite  heterogeneous  when 
one  considers  the  differences  which  exist  across  subsystems 
with  respect  to  the  number  of  programs,  average  errors/programs , 
and  average  error  rate/program.  One  major  commonality,  however, 
that  was  found  over  all  subsystems  was  the  high  intercorrela- 
tions which  existed  between  Total  Source*  Instructions  (TS) 
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and  the  variable  BR,  LS,  DATA,  NEX,  and  EX.  These  correla- 
tions are  reported  in  Table  2-7.  Since  each  of  the  variables 
within  this  group  was  highly  related  to  others  by  definition 
(i.e.,  TS+NEX  + EX,  and  EX«BR  + LS  + DATA  + other  executable 
statements  appearing  in  the  program) , the  high  intercorrelations 
were  not  surprising.  The  fact  that  these  intercorrelations 
were  consistently  high  and  of  similar  magnitude  across  all 
subsystems  is  an  interesting  finding.  It  is  not  known  whether 
this  phenomenon  can  be  explained  byt 

(1)  the  characteristics  of  the  JOVIAL  J4  programming 
language,  since  these  high  correlations  are  being 
observed  not  omly  over  eight  heterogeneous  sub- 
systems but  also  over  numerous  dissimilar  functions 
that  were  being  programmed,  or 

(2)  by  the  use  of  the  same  basic  set  of  programmers 
to  program  similar  functions  over  all  subsystems, 
or 

(3)  by  the  fact  that  this  a universal  finding,  i.e., 
one  which  applies  to  other  programming  languages 
as  well. 

In  general,  for  all  subsystems,  each  of  the  univariate 
frequency  distributions  was  highly  peaked  and  demonstrated 
minimal  to  extreme  positive  skewnesB.  The  intercorrelations 
between  each  of  the  predictors  and  errors  ranged  from  low  to 
very  high  across  all  subsystems.  Generally,  these  correlations 
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TABLE  2 7.  CORRELATIONS  OF  BR,  LS,  DATA,  NEX,  AND  EX  WITH  TOTAL  SOURCE 
INSTRUCTIONS  (TS) 
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were  higher  than  those  observed  for  the  predictor  variables 
with  errors  for  the  sample  S programs.  Also,  as  in  sample  S, 
a significant  number  of  programs  were  reported  as  error-free 
(see  Section  3.0) . 


3.0  PRELIMINARY  REVIEW  AND  ANALYSIS  OF  DATA  SAMPLES 

3.1  Selected  Limitations  of  the  Data 


The  purpose  of  this  section  is  to  briefly  enumerate  and 
discuae  selected  limitations  of  the  data  in  both  samples  that 
could  affect  or  influence  the  predictability  of  errors,  and 
resultantly  could  affect  or  limit  the  generalizability  of 
conclusions  reached  in  this  study. 

Data  Collection  and  Definition  of  Variables  - It  should 
be  clear  from  the  preceding  discussion  of  both  data  samples  in 
Section  2.0  that  serious  questions  can  be  raised  with  respect 
to  (1)  when  the  data  for  the  predictor  variables  were  collected 
vis-a-vis  the  error  data  and  (2)  the  limited  usefulness  of  the 
predictors'  definitions  and  descriptions  for  aiding  an  under- 
standing of  how  each  variable  may  uniquely  influence  or  contri- 
bute to  programming  errors.  To  be  sure,  the  need  exists  for 
future  research  projects  to  carefully  identify  and  define  the 
variables  to  be  analyzed,  discuss  why  they  were  selected,  and 
identify  what  use  is  to  be  made  of  the  data,  prior  to  the  actual 
data  collection.  With  the  definitions  presently  available, 
there  is  little  possibility  for  any  comparisons  to  be  made 
between  the  predictions  of  both  samples,  not  to  mention  the 
limited  possibility  for  comparison  of  these  predictors  with 
variables  obtained  from  other  projects  in  which  programming 
languages  other  than  CENTRAN  and  JOVIAL- J4  have  been  used. 

Classification  and  Definition  of  Errors  - For  both  data 
samples  a complete  classification  and  detailed  definition  of 
programming  error  categories  were  either  non-existent  or 
unavailable  to  be  analyzed  for  the  purposes  of  this  study. 
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Clearly  without  these  error  classifications  and  definitions 
for  each  category,  analysis  is  limited  to  an  aggregate  or  gross 
count  of  errors.  Better  predictions  might  very  well  result 
from  using  total  errors  of  a specific  type  as  the  dependent 
variable. 

Heterogeneity  Within  and  Between  Data  Samples  - Prom  the 
discussion  presented  in  Section  2.0,  it  is  apparent  that  the 
software  development  environments,  software  testing  conditions, 
programming  languages,  project  management  methods,  and  the 
command  and  control  functions  being  programmed  were  different 
in  many  respects  bstwesn  ths  two  data  samples  being  analyzed. 
Furthermore,  as  svidenced  by  the  statistics  presented  in 
Tables  2-4  and  2-6,  there  are  differences  in  variability  among 
the  three  projects  of  sample  S and  among  the  eight  subsystems 
of  sample  T.  These  differences  are  further  indicative  of  the 
functional  differences  that  existed  between  each  of  the  projects 
and  subsystems  in  the  two  samples,  and  the  individual  differ- 
ences that  existed  among  the  programmers  responsible  for  the 
software  development  effort.  Unquestionably,  these  differences 
or  lack  of  homogeneity  between  and  within  the  data  samples  will 
restrict  the  extent  to  which  the  prediction  equation  results 
for  a given  set  of  program-observations  can  be  compared  to 
other  sample  observations. 

Thoroughness  of  Program  Testing  - For  both  data  samples, 
little  is  known  about  the  thoroughness  of  testing  of  all  the 
783  programs  being  analyzed  in  this  study.  For  that  matter, 
the  prediction  equations  developed  in  this  study  are  limited 
in  that  they  apply  to  observed  errors  only.  No  information  is 
available  as  to  the  latent  errors,  or  those  which  might  be 
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found  at  a later  time  as  a result  of  more  intensive  testing 
or  operational  usage  of  the  programs.  Most  assuredly,  given 
the  increasing  manpower  and  costs  in  large-scale  command  and 
control  software  maintenance,  and  the  increasing  attention 
being  paid  to  the  relationship  between  thoroughness  of  testing 
and  software  quality  and  reliability  at  the  DOD  software 
management  level  and  in  the  research  literature,  valid  and 
reliable  measures  of  program  tastedness  need  to  be  developed 
and  applied  to  all  ongoing  and  future  software  development 
efforts. 


3.2  Preliminary  Analysis  Findings  and  Observations 


Correlations  Between  Source  Instructions  and  Other 

Predictors  - For  both  data  samples  many  of  the  predictor 
variables  had  moderate  to  high  positive  correlations  with  the 
Source  Instructions  variable  (i.a.,  XI  for  sample  S,  and  TS 
for  sample  T) . For  example,  for  project  M of  sample  S which 
involved  395  of  the  total  534  programs  being  analyzed  for  this 
sample,  30  of  the  53  predictors  had  correlations  which  ranged 
from  .30  to  .99.  Similar  correlations  were  also  observed  for 
projects  B and  P of  this  sample.  For  subsystem  A of  sample  T 
which  had  the  largest  number  of  program-observations  (N«51) 
of  any  subsystem,  11  of  the  15  predictors  had  correlations  which 
ranged  from  .37  to  .99.  There  too,  similar  correlations  were 
observed  over  each  of  the  remaining  subsystems,  B thru  H.  In 
fact  the  consistency  of  some  of  these  correlations  for  the  BR, 

LS,  DATA,  NEX,  and  EX  variables  over  the  eight  subsystems  was 
reported  earlier  in  Table  2-7.  In  general  for  both  data  samples, 
many  predictors  were  also  correlated  to  a similar  degree  with 
other  predictors  besides  Source  Instructions.  However  none  of 
these  predictors  was  correlated  over  the  large  number  of  vari- 
ables with  the  same  magnitude  as  was  Source  Instructions. 

Generally,  when  many  highly  intercorrelated  variables  are 
being  used  for  prediction  purposes,  serious  mathematical  prob-* 
lems  result  (e.g.,  the  matrix  of  intercorrelations  among  pre- 
dictors may  become  singular) , which  yield  an  indeterminate 
solution  to  the  prediction  equation.  For  this  reason,  many  of 
the  variables  in  both  samples  having  very  high  correlations 

i 

with  Total  Source  Instructions  were  eliminated  frojm  the  analysis. 
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For  those  variables  which  remained;  i.e. , those  that  did 
not  correlate  very  highly  with  Source  Instructions,  it  was 
desirable  to  obtain  an  additional  measure  of  their  contribu- 
tion (or  correlation)  to  programming  errors  with  the  effect 
of  Total  Source  Instructions  removed.  This  consideration  leads 
directly  to  the  need  to  "normalize"  the  predictor  variables. 

Basically,  the  effect  of  source  instructions  was  to  be 
removed  from  each  of  the  predictor  variables  by  means  of  the 
normalization  procedure.  Although  several  more  involved  compu- 
tational procedures  are  available  as  alternatives  for  doing 
this,  it  was  decided  to  divide  each  predictor's  value  in  a 
given  program  by  the  number  of  Source  Instructions  for  that 
program. 

As  an  end  result  of  this  normalizing  procedure  being 
applied,  (1)  a net  doubling  occurred  to  the  number  of  predic- 
tor variables  that  could  be  considered  in  any  one  prediction 
equation  for  each  sample,  and  (2)  a new  dependent  variable 
(errora/source  instructions)  was  added  to  each  program-observa- 
tion which  is  referred  to  as  the  error  rate  per  program. 

Error  Rate  and  Length  of  Program  - Once  the  normalization 
procedure  had  been  carried  out  and  the  correlations  among  all 
variables  were  once  again  obtained,  it  was  observed  with  interest 
that  the  correlation  of  error  rate  (i.e.,  the  normalized  errors 
per  program  variable)  with  Source  instructions  was  negative, 
and  low  to  moderate  in  magnitude,  over  most  of  the  samples  to 
be  analyzed.  Interpreting  the  correlations  directly  meant 
that  as  the  number  of  source  instructions  in  a program  increases, 
the  errors  per  100  lines  of  code  decreases,  and  vice-versa. 
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In  essence,  for  the  programs  in  the  two  samples  this  suggested 
that  the  shorter  programs  have  higher  error  rates  than  the 
longer  programs.  Nevertheless,  when  the  relationship  between 
error  rate  and  source  instructions  was  actually  graphed  for 
projects  M,  B,  and  P of  sample  S,  as  presented  in  Figure  3-1, 
the  reason  for  the  low  negative  correlations  became  more 
apparent . 

These  graphs  show  that,  for  each  of  the  three  projects, 
as  number  of  source  instructions  increases  the  error-rate 
increases,  reaching  a maximum  error-rate  in  the  range  of 
200-400  source  instructions.  From  that  point  on,  the  error- 
rate  decreases  as  number  of  source  instructions  increases. 

This  phenomenon  requires  some  explanation. 

It  is  well-known  that  as  the  number  of  source  instructions 
increases,  the  number  of  possible  paths  through  the  program 
usually  increases  and  that  this  increase  is  at  a more  rapid 
rate  than  a linear  one  (perhaps  not  exponential,  but  more  than 
linear) . To  detect  the  same  percentage  of  total  errors  in 
two  programs,  the  testing  effort  should  exercise  approximately 
the  same  percentage  of  total  paths.  Therefore,  the  amount  of 
testing  to  detect  equal  percentages  of  total  errors  should 
increase  at  a rate  faster  than  linearly.  It  is  hypothesized 
that  such  was  not  the  case  for  these  projects,  and  that  the 
negative  correlation  between  length  of  program  and  error  rate 
is  due  to  lack  of  thoroughness  of  testing.  That  is,  the  shorter 
programs  were  more  thoroughly  tested  than  the  longer  ones,  in 
terms  of  having  a higher  percentage  of  their  paths  executed. 


FIGURE  3-1.  ERROR  RATE  AND  SOURCE  INSTRUCTION 

RELATIONSHIP  FOR  PROJECTS  M,  B,  AND  P 
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It  is  o£  dourse  possible  that  other  explanations  (or 

if  . 

hypotheses)  would  be  better , such  as,  for  these  projects  (include 
ing  both  samples)  the  shorter  programs  were  in  fact  more  com- 
plex, or  that  longer  programs  contain  in  general  more  repeti- 
tive operations  (thus  more  easily  programmed) , but  it  is 
proposed  here,  as  a result  of  evaluating  the  total  analysis 
effort,  that  the  hypothesis  of  inadequate  testing  of  the 
longer  programs  is  more  likely  to  represent  the  actual  situation. 

Programs  With  Zero  Reported  Errors  - Throughout  most  of  the 
sets  of  program  samples  that  were  to  be  analyzed,  zero  reported 
errors  were  observed  in  long  as  well  as  short  programs.  Statis- 
tics on  these  zero  reported  errors  programs  were  reported 
earlier  in  Table  2-4  for  the  projects  of  sample  S and  in 
Table  2-6  for  the  subsystems  of  sample  T.  For  the  longer  pro- 
grams having  zero  reported  errors,  some  skepticism  is  warranted. 
It  could  have  been  that  programming  errors  went  unreported 
for  these  programs,  or  that  the  programs  received  very  little 
testing.  For  the  shorter  programs  with  zero  errors,  it  was 
considered  that  they  could  actually  be  error  free  as  reported. 
However,  given  the  earlier  observed  relationship  between  error 
rate  and  source  instructions  as  indicated  by  Figure  3-1  and 
the  correlations  that  were  obtained,  it  is  suggested  that  at 
least  some  of  these  programs  underwent  minimal  testing. 

It  is  believed  then,  that  programs  reported  as  error  free 
constitute  a set  of  programs,  some  of  which  are  actually  error 
free  and  some  of  which  contain  an  unknown  number  of  errors. 
Further,  those  reportedly  error  free  programs  are  more  likely 
to  have  more  latent  errors  than  those  programs  with  some  num- 


3-8 


r •, 


■ < **+  — 1 


ber  of  errors  reported.  (The  same  reasoning  might  also  be 
applied  to  those  programs  with  only  one  or  two  reported 
errors;  however,  the  line  must  be  drawn  somewhere). 


f- 

(*  Throughout  the  analysis  for  both  errors  and  error  rate, 

results  were  obtained  (1)  leaving  the  zero  error  programs  in 
the  analysis,  and  also  (2)  excluding  these  programs.  Per- 
forming  the  analysis  in  two  ways,  it  was  possible  to  determine 
whether  an  increase  in  the  predictability  of  errors  would 
result  by  eliminating  one  source  of  ambiguity  in  the  data. 

At  best,  performing  analysis  in  this  way  would  be  able  to  do 
justice  to  any  researchers  who  would  contend  that  if  error 
prediction  equations  being  developed  are  to  be  effective  at 
predicting  errors,  then  all  programs  used  to  develop  these 
equations  should  have  errors  reported  in  them. 
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4.0  ANALYSIS  METHOD  AND  PROCEDURE 


4.1  Multiple  Linear  Regression  Analysis 


The  method  of  analysis  used  in  this  study  to  predict 
programming  errors  was  that  of  multiple  linear  regression. 
Using  the  model/ 


yi  " a + blxli  + b2X2i  + 


' * bnXni 


(4.0) 


the  observed  programming  errors  (Y^)  were  assumed  to  be 
predictable  from  a linear  combination  of  the  program  charac- 
teric  predictor  variables  (X^,  x2,  ...  xn)  . In  (4.0), 
bl'  b2'  * - ' bn  are  tbe  regression  coefficients;  i.e.,  the 
estimated  weights  for  each  of  the  predictors  that  will  maxi- 
mize the  predictability  of  errors,  a is  the  intercept  constant, 


a - Yi  - + b2*2i  + •••  + bn5fni) 


(4.1) 


i.e.,  the  estimated  value  of  Y^  at  the  point  where  the  regres- 
sion hyper-plane  crosses  the  Y axis,  and  Y*  represents  the 
predicted  or  estimated  value  of  errors  for  each  individual 
program  module  (i)  . The  quantities  7 and  are  mean  values 
of  the  respective  variables. 

The  method  used  to  determine  the  parameters  of  the  re- 
gression equation  is  to  minimize  the  sum  of  squared  deviations 
of  actual  errors  from  predicted  errors;  i.e,,  minimize 

5 ’ 'Zlfi  • (**blxU  + b2*u  + + »nxnlj]  2 (4-21 

s - 2(vi  • *i)2  ' (B-3> 


In  (4.3)  the  value  of  S is  referred  to  as  the  sum  of 
squares  of  deviations  of  the  estimated  value  (Y^)  of  errors 
from  the  observed  value  (Y^)  of  errors  summed  over  all  the 
program  modules  in  each  sample  or  set  of  observations  for 
which  errors  are  predicted. 

Generally  when  prediction  equations  such  as  represented 

in  (4.0)  are  being  evaluated  for  their  goodness  of  prediction 

two  statistics,  (1)  the  multiple  correlation  coefficient  (R) 

3 

and  (2)  the  squared  multiple  correlation  coefficient  (R  ) , 
are  used.  The  value  of  R has  a range  from  0 to  1 as  indicated 
in  (4.4)  , 


0 ^ Ry . 1 2 3 . . .n  £ 1 (4, 

and  can  be  interpreted  as  the  actual  correlation  between  the 

linear  combination  of  predictor  variables  and  the  observed 

2 

values  of  errors  ( Y^) . The  value  of  R also  has  a range  from 

0 to  1 and  is  a measure  of  the  proportion  or  percentage  of 

variation  in  the  dependent  variable  (Y^)  that  can  be  accounted 

for  or  explained  by  the  linear  combination  of  predictors. 

2 

More  specifically  R can  be  represented  by  the  ratio, 

RJ.123...n  -2>1  - 7 )2/SYi  - 7,2  14- 

where  ? is  the  mean  or  average  value  of  the  dependent  variable 
in  the  given  sample  being  analyzed. 

As  more  improved  or  better  predictions  are  obtained,  the 

2 

valuea  of  both  R and  R will  approach  a value  of  1.  Complete 

lack  of  correlation  (implying  no  predictability)  results  in  an 

2 2 
R or  R value  of  zero.  Both  statistics  (R  and  R ) will  be 
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presented  for  all  prediction  equations  in  this  study.  For  the 

purposes  of  this  report,  the  goodness  of  prediction  or  what 

could  be  termed  the  predictability  of  programming  errors  will 

2 

be  discussed  only  in  terms  of  R . 

Predictability  will  be  described  in  terms  of  being  low, 
moderate,  or  high  in  later  sections  of  this  report.  These 
categories  are  based  on  the  authors'  judgement  and  are  used 
for  the  descriptive  purpose  of  comparing  and  contrasting  the 
results  of  numerous  error  predictions  obtained  in  this  study. 

For  those  readers  interested  in  a more  thorough  and 
detailed  discussion  of  regression  analysis,  texts  by  Draper 
and  Smith  (1968)  (3)  and  Kerlinger  and  Pedhazur  (1973)  (6) 
are  readable,  cogently  presented,  and  are  highly  recommended . 

Although  much  more  could  be  said  here  concerning  multiple 
regression  theory,  practice,  and  procedures,  the  following  two 
points  deserve  special  emphasis. 

Multicollinearity  Among  Predictor  Variables  - In  many 
multiple  regression  procedures  the  computed  values  of  a,  b^, 
b- , ...  and  b are  directly  obtained  from  matrices  which  con- 
tain  tiie  intercorrelations  among  the  predictor  variables  and 
the  correlations  of  the  predictor  variables  with  errors.  If 
the  predictor  variables  are  truly  mathematically  independent 
(i.e.,  not  correlated  with  each  other),  then  the  problem  of 
interpreting  the  final  multiple  regression  equation  becomes 
simple  and  the  interpretation  is  straightforward.  However, 
when  the  predictor  variables  are  moderate  to  highly  correlated 
with  each  other  (which  is  usually  the  case) , then  a clear  and 
explicit  interpretation  of  the  prediction  equation  becomes 


comparatively  much  more  difficult.  When  this  situation 

arises,  then  the  problem  of  multicollinearity  is  said  to  exist 
(Althauser,  1971  ; Gordon,  1968;  Rama  Sastry,  1970)  (1,  5,  15). 
As  indicated  in  Sections  2.0  and  3.0,  many  of  the  variables 
in  both  of  our  samples  were  moderately  to  highly  correlated. 
Necessarily  then  in  the  'Results'  section  of  this  report 
(Section  6.0)  , our  interpretation  of  the  prediction  equations 
will  consider  interdependent  variables. 

v I 

Number  of  Predictor  Variables  and  Sample  Size  - When  the 
number  of  predictor  variables  available  to  study  equals  or 
exceeds  the  number  of  program-observations  in  the  data  sample, 
then  perfect  prediction  of  the  criterion  error  measure  (errors 
or  error  rate)  will  always  result.  Clearly  for  both  data 
samples,  there  is  a danger  of  encountering  this  problem.  What 
was  needed  then  to  deal  with  this  problem  was  to  apply  some 
analytical  or  automated  statistical  procedure  to  select  the 
most  powerful  set  of  predictors  for  errors  and  error  rate  in 
each  of  the  data  samples.  Two  procedures  were  used.  The  first 
procedure  involved  the  a priori  elimination  of  predictor  vari- 
ables based  on  selected  operational  and  statistical  criterion. 
The  criteria  and  the  variables  eliminated  using  these  criteria 
are  discussed  in  Section  5.0.  The  second  procedure  involved 
the  use  of  a stepwise  regression  procedure  that  would  enter, 
remove,  swap,  or  delete  variables  from  the  regression  equation 
based  totally  upon  statistical  criterion.  This  procedure  will 
be  discussed  in  the  following  section. 
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BMDP  Stepwise  Regression  Procedure 


The  stepwise  regression  procedure  selected  to  be  used  in 
this  study  was  tho  procedure,  referred  to  as  BMDP2R,  made 
available  along  with  other  statistical  programs  in  the  Bio- 
medical  Computer  Program  Package  (BMDP)  developed  and  distri- 
buted by  the  Health  Sciences  Computing  Facility  (HSCF)  at  the 
University  of  California,  Los  Angeles  (UCLA) . The  HSCF 
supports  Biomedical  computer  analysis  for  the  School  of 
Medicine  at  UCLA  and  is  sponsored  by  a NIH  Special  Research 
Resources  Grant  RR-3.  Although  the  BMDP  series  of  programs 
has  only  been  recently  released  (1975) , it  is  based  on  an 
entire  series  of  programs  that  have  had  a history  of  program 
revisioning,  maintenance,  modification,  and  testing  that  goes 
back  to  1961.  The  BMD  program  series  is  one  of  the  most 
widely  used  and  highly  reliable  sets  of  statistical  programs 
that  has  yet  been  developed  for  general  purpose  applications. 

Essentially,  the  BMDP 2 R procedure  estimates  the  parameters 
(a,  b^,  b2,  ...  bn)  of  multiple  linear  regression  equations 
in  a stepwise  manner.  Four  stepping  algorithms  are  available. 
The  algorithm  selected  for  the  current  analysis  is  referred 
to  as  an  FSWAP  procedure,  which  enters  and  removes  predictor 
variables  based  on  an  F-to-enter  and  a F-to-remove  criterion 
with  a variable  interchange  option.  Without  becoming  too 
detailed  here,  the  FSWAP  procedure  works  as  follows.  Initially, 
the  procedure  selects  the  variable  having  the  highest  correla- 
tion with  the  dependent  variable.  It  then  adds  the  variable 
which  has  the  next  highest  partial  correlation  with  V.  After 
this  addition,  it  continues  adding  variables  which  increase 
the  multiple  correlation  coefficient  ( R) , and  concurrently 
swapping  or  exchanging  variables  not  yet  in  the  equation  which 
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increase  the  multiple  R value.  Finally  after  no  additional 
variables  will  increase  the  value  of  R,  it  then  proceeds  to 
perform  a backwards  elimination,  removing  those  variables 
which  have  the  least  effect  on  the  goodness  of  prediction. 

For  additional  information  on  this  program,  the  reader  should 
consult  the  actual  BMDP  publication  manual  (Dixon,  1975) 

(2).  (All  computer  runs  using  the  BMDP 2 R program  ware  per- 
formed on  an  IBM  370/168  computer.) 
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5.0  DEVELOPMENT  OF  ERROR  PREDICTION  EQUATIONS 

The  purpose  of  this  section  is  to  discuss  various  opera- 
tional and  analytical  considerations  and  decisions  that  were 
made  which  pertain  to  the  development  of  the  error  prediction 
equations  that  were  obtained  for  this  study. 

5.1  Normalization  of  Predictor  Variables 

For  both  data  samples,  various  transformations  of  selected 
variables  were  required.  For  example,  for  each  program  in 
both  sample  S and  sample  T,  in  order  to  normalize  the  original 
predictor  variable  values,  each  value  was  multiplied  by  a 
constant  of  100  and  divided  by  the  total  number  of  source 
instructions  for  that  particular  program  (i.e.,  either  XI 
for  sample  S programs  or  TS  for  sample  T programs) . This 
normalization  procedure  thus  resulted  in  a completely  new 
set  of  predictor  variables,  in  addition  to  the  original  set 
of  predictors,  in  which  each  normalized  variable  could  be 
interpreted  as  a unique  measure  of  the  original  predictor 
variable  given  that  the  effect  of  the  length  of  the  program 
had  been  removed  from  it.  In  effect  this  procedure  made  each 
of  the  original  variables  comparable  with  respect  to  a "per  100 
lines  of  source  code"  interpretation. 

It  is  interesting  to  note  that  this  particular  normali- 
zation procedure  did  not  completely  eliminate  the  linear 
effect  of  total  source  instructions  from  all  of  the  original 
variables.  For  most  of  the  normalized  variables,  the  corre- 
lation coefficients  with  Total  Source  Instructions  were  nega- 
tive but  relatively  low  or  close  to  zero  in  magnitude.  A few 
of  the  normalized  variables  had  non-zero  correlations  with  Total 
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Source  Inetructionc  which  ranged  from  -.50  to  +.40  in 
sample  S and  from  -.69  to  +.72  in  sample  T.  Clearly  then , 
this  normalization  procedure  is  not  the  most  effective  means 
for  eliminating  the  effect  of  Source  Instructions  from  each  of 
the  predictor  variables.  In  actuality , the  most  statistically  - 
consistent  and  accurate  procedure  for  eliminating  the  linear 
effect  of  one  variable  from  one.  or  a set  of  other  variables 
is  through  partial  correlation. 

In  the  case  of  this  analysis  effort,  use  of  the  partial 
correlation  procedure  could  have  been  alternatively  employed 
by  first  predicting  each  original  variable  individually  as  a 
function  of  Total  Source  Instructions,  and  then  using  only  the 
residual  values  (i.e.,  Y^-Y^)  as  the  values  for  the  normalized 
variables  that  were  employed  in  the  prediction  equations.  If 
this  procedure  had  been  utilized,  the  resulting  matrix  of 
intarcorrelations  among  all  the  normalized  variables  would  in 
fact  be  a matrix  of  partial  correlations;  that  is,  the  corre- 
lation between  the  various  pairs  of  predictor  variables  with 
the  linear  effect  of  Total  Source  Instructions  eliminated  from 
both  variables.  The  matrix  of  intercorrelations  between  each 
of  the  normalized  variables  and  the  original  variables  including 
the  dependant  variable,  errors,  would  actually  be  a matrix  of 
semi -partial  correlations;  i.e.,  the  correlations  between  each 
of  the  normalized  variables  and  the  original  variables  including 
the  dependent  variable  with  the  linear  effect  of  Total  Source 
Instructions  eliminated  from  only  the  normalized  variables. 

Clearly,  one  implication  of  using  this  alternate  procedure 
is  that  for  the  normalized  variables,  an  entirely  new  data  base 
must  be  constructed  wherein  each  residual  value  must  first  be 
obtained  from  a simple  linear  regression  procedure,  automated, 
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and  then  entered  in  the  data  base.  To  be  sure,  constructing 
normalized  variables  using  this  procedure  would  indeed  be 
time  consuming  and  would  require  the  availability  of  special- 
purpose  software  support  programs  that  could  minimize  the  amount 
of  manual  processing  that  would  be  required.  Due  to  time 
constraints,  this  procedure  was  not  followed. 

5.2  Other  Transformations  to  Predictor  Variables 


In  addition  to  the  normalization  that  was  applied  to  the 
predictor  variables  for  both  samples,  the  only  other  transfor- 
mations that  were  used  were  applied  against  several  predictors 
of  sample  T.  Since  all  data  from  sample  T had  been  obtained 
in  manual  form  for  this  study,  all  data  values  had  to  be 
recorded,  keypunched,  verified,  and  then  automated.  Most  pre- 
dictor variables  of  sample  T were  whole  numbers,  whereas  the 
Loop  Complexity  (LL) , If  Complexity  (IF),  Programmer  Rating 
(RAT) , and  Programmer  Workload  (WKLD)  variables  were  whole 
numbers  with  one  decimal  value.  In  addition,  Comments  (COM) 
and  Programmer  Rating  were  originally  represented  as  negative 
values.  In  order  to  simplify  and  expedite  the  manual  to  auto- 
mated process  for  this  data,  the  values  of  LL,  IF,  RAT,  and  WKLD 
were  multiplied  by  a value  of  10  in  order  to  represent  them  as 
whole  numbers,  The  values  of  COM  and  RAT  were  multiplied  by 
-1  and  -10,  respectively,  in  order  to  eliminate  the  negative 
sign  from  both  variables  and  represent  the  RAT  predictor 
values  as  whole  numbers. 

Additionally,  using  the  RAT  and  WKLD  variables  of  sample  T, 
a new  predictor  variable  was  constructed  which  took  the  form  of 
RAT/WKLD . It  was  hypothesized  that  this  new  variable  was 
linearly  related  to  errors  and  as  such  should  be  included  as 
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a candidate  variable  for  consideration  by  the  stepwise 
regression  procedure.  This  newly  constructed  value  was  multi- 
plied by  a constant  of  100  in  order  to  represent  it  as  a whole 
number  throughout  its  range  of  values. 
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5 . 3 Combining  Predictors  In  The  Equations 


As  a direct  result  of  the  normalization  procedure,  107 
distinct  variables  versus  the  original  54  variables  of  sample  S 
were  considered  as  predictors  of  errors  and  error  rate.  For 
sample  T a total  of  30  as  opposed  to  the  original  16  variables 
were  candidate  predictors.  Although  the  BMDP2R  program  is 
a stepwise  procedure  that  selects  the  optimal  predictors  from 
among  all  those  available  to  be  entered,  it  is  desirable  to 
limit  the  number  of  predictors  it  would  have  to  consider  for 
any  given  set  of  predictions . This  was  desired  in  order  to 
maximize  the  chance  that  as  many  predictors  as  possible  could 
be  considered  in  the  regression  equation  simultaneously,  and 
then  eliminated  one  at  a time  using  the  backwards  elimination 
procedure  if  the  variable  actually  had  no  significant  effect 
on  the  predictability  of  the  dependent  variable.  In  addition, 
the  predictors  available  for  selection  by  the  regression  pro- 
cedure are  limited  because  the  total  number  of  predictors 
should  not  exceed  the  sample  size.  In  addition,  reducing  the 
number  of  predictors  reduces  the  computer  time  required  to 
generate  each  set  of  predictions. 

For  the  sample  T subsystems,  allowing  all  30  or  less  vari- 
ables to  be  available  for  selection  in  the  equation  presented 
no  major  d.i.f:  Lrulty . Due  to  the  a priori  elimination  of  pre- 
dictors that  was  carried  out  for  sample  T (to  be  discussed  in 
Section  5.6),  no  more  than  23  variables  were  ever  allowed  to 
be  considered  fcr  selection.  Essentially  then  for  each  sub- 
system of  sample  T,  errors  and  error  rate  were  predicted  using 
the  combination  of  variables  as  follows: 
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Errors/program  ■ f (Program  Structure  + Programmer  Variables) 

Errors/program  » f (Program  Structure  Variables  only) 

Error  rate/program  = f (Program  Structure  + Programmer 

Variables) 

Error  rate/program  - f (Program  Structure  Variables  only) 


The  program  structure  variables  for  these  predictions  represen- 
ted  the  combination  of  both  the  unnormalized  (TS,  LL,  IP,  BR, 

. COM)  and  normalized  (LL/TS,  IF/TS , BR/TS,...,  COM/TS) 
predictor  variables.  Additionally,  prediction  equations  were 
obtained  first  using  the  program  observations  available  in 
each  subsystem  and  then  second,  using  only  the  remaining 
program-observations  left  after  the  zero  reported  error  programs 
had  been  deleted  from  the  analysis. 

For  the  three  projects  of  sample  S,  the  107  predictors  were 
analyzed  in  two  different  sets  each,  for  errors  and  error  rate. 
These  sets  of  predictors  were  combined  as  follows: 


Errors/program  = f (Unnormalized  Variables) 
Errors/program  ■ f (SI  + Normalized  Variables) 
Error  rate/program  =*  f (Unnormalized  Variables) 
Error  rate/program  = f(Sl  + Normalized  Variables) 


In  these  predictions  SI  represents  the  Source  Instructions  (SI) 
variable  XI ; the  unnormalized  variables  are  the  predictors 


XI,  X2,  X54;  and  the  normalized  variables  are  the 

predictors  X56,  X57,  X107.  Here,  as  with  sample  T, 

prediction  equations  were  obtained  both  using  all  observations 
and  using  only  the  observations  remaining  after  tne  zero 

reported  error  programs  were  deleted. 

In  any  of  these  equations,  due  to  the  a priori  variable 
elimination  procedure  that  was  applied,  no  more  than  45  pre- 
dictors per  regression  run  were  ever  available  to  be  selected 
for  the  sample  S prediction  equations. 


5-7 


5.4  Selection  of  Regression  Coefficients  to  be  Reported 


After  having  made  the  necessary  transformations  (i.e., 
normalization  and  other  transformations  required)  to  the  data 
and  identifying  how  the  predictors  were  to  be  combined  in  the 
analysis , one  other  issue  remained.  In  essence,  there  were 
sets  of  predictors  in  both  samples  that  would  be  equally  or 
unequally  weighted  and  non-comparable  in  a regression  analysis. 
This  follows  from  the  facts  that  1)  the  normalized  variables 
had  different  units  of  measurement  as  compared  with  the  unnorm- 
alized variables,  and  2)  for  the  predictors  uaed  in  sample 
T,  the  program  structure  and  programmer  variables  were  not 
comparable,  being  derived  from  two  distinct  measurement  domains. 
Thus,  using  predictors  that  had  unequal  or  non -comparable  units 
of  measurement  would  result  in  making  more  difficult  any  rela- 
tive comparisons  among  the  raw  regression  coefficients  computed 
for  the  predictors  in  the  equation.  In  order  to  resolve  this 
problem,  it  was  decided  that  the  standardized  partial  regression 
coefficients  (Kerlinger  and  Pedhazur,  1973,  p.  64)  (8)  would  be 
reported  for  all  predictions . 
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Regression  Analysis  Using  Standardized  Form  of  the 
Prediction  Equation 

In  practice  the  standardized  partial  regression  coefficients 
are  referred  to  as  B (beta)  coefficients  or  beta  weights  as 
compared  to  the  b coefficients  as  represented  in  equation  (4.0). 
The  beta  weights  are  the  regression  coefficients  that  result 
when  the  raw  data  is  transformed  (i.e.,  standardized  in  this 
case)  into  standard  score  form  prior  to  the  analysis.  For 
example , the  standard  score  for  the  ith  observation  on  a 
variable  (X^)  is  computed  as  follows: 

«i-  Xi-^  (5.0) 


where  X^  and  s^  are  the  mean  and  standard  deviation,  respec- 
tively, for  that  variable.  When  all  the  predictor  and  depen- 
dent variables  are  standardized  according  to  this  procedure, 
the  standardized  variables  all  have  a mean  of  0 (i.e,,  Z ^ - 0) 
and  a standard  deviation  of  1 (i.e.,  s2  - 1) . Essentially 
then,  the  variability  in  each  variable  is  made  comparable 
with  respect  to  the  standard  deviation  as  the.  common  unit 
of  measurement  over  all  variables;  thus,  the  B coefficients 
in  the  standard  score  form  of  the  regression  equation  are 
also  comparable.  Although  the  raw  data  had  not  been  stand- 
ardized, the  BMDP2R  regression  procedure  computes  the  beta 
coefficients  (in  addition  to  the  raw  b coefficients) . 

Since  the  standard  score  form  of  the  regression  equation 
was  being  reported  in  this  analysis,  the  linear  regression 
model  and  other  statistical  formula  became  more  easily  interpre- 
ted in  terms  of  the  beta  coefficients.  For  example,  the 


linear  model  now  took  the  form, 


Zyi  “ BlZli  + B2Z2i  + B3Z3i  + •••  + BnZni 


(5.1) 


2 

and  the  multiple  R,  R , and  standard  error  of  estimate  s^ 
could  be  computed  as  follows: 


Ry.l23...n  lryl  + B2ry2  + B3ry3  + •••  + Bnryn 

Ry.l23...n  “ Blryl  + B2ry2  + B3ry3  + * * * + Bnryn 


V1  ” Ry  • 123 , . .n 


(5.2) 

(5.3) 

(5.4) 


where  the  r ^ values  are  the  correlations  of  each  predictor  (i) 
with  the  dependent  variable  (y) . 


Furthermore,  since  the  results  of  any  correlational 
analysis  are  the  same  whether  the  analysis  started  with  the 
raw  data  values  or  the  standard  score  values,  computational 
formula  are  available  (Kerlinger  and  Pedhazur,  1973,  pp.  61-62) 
(8)  which  easily  allow  the  computation  of  the  value  of  the  b 
coefficients  and  the  a intercept  in  equation  (4.0)  using  the 
beta  coefficients  obtained  in  this  analysis.  For  example, 
the  raw  regression  coefficients  bj  and  the  a intercept  can  be 
directly  computed  using  the  following  formula: 


b 


j 


(5.5) 
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and 


(5.6) 


where  and  are  the  standard  deviations  of  the  dependent 
(v)  and  predictor  variables  (j)  , respectively,  and  7 and  5?^  are 
the  means  for  the  dependent  and  predictor  variables,  respec- 
tively. 
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5.6  A Priori  Elimination  of  Predictor  Variables 


Certain  predictor  variables  in  both  data  samples  were 
eliminated  from  any  further  consideration  in  the  prediction 
analysis , prior  to  their  actual  consideration  for  selection 
by  the  BMDP2R  regression  procedure.  In  general,  most  of  the 
predictors  that  were  eliminated  at  this  early  stage  of  analysis 
were  done  so  in  order  to  reduce  the  incidence  of  multi- 
collinearity  that  exists  among  the  predictors.  Other  variables 
were  eliminated  primarily  because  they  had  zero  values  through- 
out the  data  sample.  Table  C-l  in  Appendix  C lists  all  the 
107  predictors  of  errors  and  error  rate  in  sample  S and  identi- 
fies each  of  the  variables  that  were  either  eliminated  a priori 
from  the  analysis,  or  made  available  to  be  considered  for 
selection  in  the  regression  procedure.  The  criteria  used  to 
eliminate  these  variables  prior  to  the  regression  analysis  are 
enumerated  at  the  end  of  this  table.  For  the  30  predictors 
of  sample  T,  Tables  C-2  and  C-3  in  Appendix  C provide  similar 
information  for  the  predictors  of  errors/program  and  error 
rate/program,  respectively. 
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ERROR  PREDICTION  EQUATIONS: 


RESULTS  AND  DISCUSSION 


The  purpose  of  this  section  is  to  present  and  discuss 
the  actual  prediction  equation  results  that  were  obtained 
when  predicting  errors/program  and  error  rate /program*  re- 
spectively! for  each  of  the  three  projects  of  sample  S 
and  each  of  the  eight  subsystems  of  sample  T.  Preliminary 
to  this  presentation  of  results  for  each  sample!  an  overall 
general  summary  of  results  with  discussion  is  provided. 

6.1  Results  Summary 

The  following  are  the  major  results  obtained  regarding 
the  predictability  of  errors  and  error  rate  over  both  of 
the  samples  that  were  analyzed. 

Errora/Program 

• For  sample  T,  where  the  error  data  had  been  collected 
throughout  the  validation!  acceptance,  integration, 
and  operational  testing  phases  of  software  system 
development,  errors/program  were  found  to  be  moderately 
to  highly  predictable.  This  predictability  was  far 
from  perfect  with  76X  to  93%  of  the  variance  accounted 
for  when  the  errors/program  were  predicted  from  a 
linear  combination  of  program  structure  variables. 

e For  sample  S,  where  the  error  data  had  been  collected 
only  during  the  test  and  integration  phase  of  software 
system  development,  errors/program  were  found  to  be 
less  consistently  predictable.  The  percent  of  variance 
accounted  for  in  this  sample  was  59%  to  9 0 Jt . 
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• Predictor  variables  which  reflected  length  of  program 
were  generally  found  to  be  the  best  single  predictors 
of  errors/program.  However,  other  program  structure- 
complexity  variables  together  in  combination  with 
length  of  program  variables  contributed  significantly 
to  the  predictability  of  errors/program. 

Error  Rate/Program 

• For  sample  T,  error  rate/program  was  found  to  be  less 
predictable  in  general  than  arrors/program,  with  59 *8  to 
8 5 ft  of  the  variance  accounted  for,  when  predicted  from 
a linear  combination  of  program  structure  variables. 

e For  sample  S,  the  predictability  of  error  rate/program 
was  generally,  lower,  with  3 4 fS  to  9 4 B of  the  variance 
accounted  for. 

• Predictor  variables  which  ware  measures  of  the  number 
of  program  interfaces  per  100  lines  of  source  code 
wore  generally  found  to  be  the  best  single  predictors 
of  error  rate/program.  However,  other  normalized 
measures  of  program  complexity  together  in  combination 
with  program  interfaces  per  100  lines  of  source  code, 
contributed  significantly  to  the  predictability  of 
error  rate/program. 
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• In  general , the  predictor  variables  v>hich  were  moat 
frequently  selected  by  the  stepwise  regression  pro- 
cedure as  contributing  significantly  to  the  predicta- 
bility of  error  rate/program  were  the  normalized 
variables . 

e Analysis  of  the  error  rate/progran  measure  for  those 
data  samples  having  a high  percentage  of  error-free 
programs  leads  to  a clear  indication  of  the  lack  of 
thoroughness  of  testing  in  these  reportedly  error 
free  programs. 

6.2  Discussion 


Clearly,  the  results  obtained  from  the  analysis  of  errors 
and  error  rate  are  not  surprising.  The  facts  that  (1)  length  of 
program  and  the  number  of  program  interfaces  per  100  lines  of 
source  code  were  found  to  be  the  best  single  predictors  for 
errors  and  error  rate,  respectively,  and  that  (2)  program  com- 
plexity variables  contributed  significantly  to  the  predictability 
of  each  dependent  variable,  are  findings  that  not  only  appeal  to 
experience  and  intuitive  judgement  about  how  these  predictor 
variables  may  be  related  to  measures  of  programming  errors, 
but  also  are  findings  which  are  supported  by  other  empirical 
studies  concerned  with  software  reliability  (Mitchell  et  al., 
1976;  Thayer  et  al.,  1976,  Okimoto,  1975) (11,  13,  14). 

For  example,  consider  the  following  hypotheses,  which 
logically  follow  from  our  knowledge  and  experience  of  program- 
ming, which  concern  the  effect  of  increasing  program  length 
and  program  complexity  on  the  total  number  of  programming  errors 
in  the  program. 
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Hypothesis  1:  As  length  of  program  increases,  program  complexity 

increases,  and  the  number  of  latent  errors  in 
the  program  increases  linearly  as  a function  of 
both. 


Hypothesis  2:  As  length  of  program  increases,  program  complexity 

increases,  and  the  number  of  probable  or  latent 
errors  in  the  program  increases  at  an  exponential 
rate. 

Hypothesis  3:  As  length  of  program  increases,  program  complexity 

increases;  redundancy  in  the  use  of  similar 
software  functions  and  instructions  in  the 
program  also  increases,  resulting  in  the  number 
of  latent  errors  in  the  program  increasing  up 
to  a point  with  no  significant  increase  there- 
after with  increasing  program  length. 


Each  of  these  three  hypotheses  are  graphically  depicted  in 
Figure  6-1. 


Hypothesis  1,  which  in  essence  is  the  basic  assumption  of 
the  multiple  linear  regression  model  used  in  this  analysis, 
can  generally  be  accepted  as  one  explanation  for  the  high  degree 
of  predictability  obtained  when  predicting  errors/program  in 
this  study.  However,  hypotheses  2 and  3 cannot  be  rejected 
by  this  analysis.  This  is  clear  for  several  reasons: 

(1)  the  model  investigated  by  this  analysis  was  a linear 
model,  and  not  an  exponential  or  curvilinear  model 
as  are  suggested  by  hypotheses  2 and  3,  respectively; 
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(2)  the  linear  model  did  not  yield  perfect  predictability 
of  errors  over  each  of  the  samples  that  were  analyzed? 
and 

(3)  the  numerous  program  structure  and  complexity  metrics 
that  were  used  as  predictor  variables  in  the  study 
would  possibly  require  re-evaluation  or  re-formulation 
with  the  chance  that  some  variables  would  be  ex- 
cluded and  new  ones  included,  in  order  to  use  them 

to  appropriately  investigate  hypotheses  2 and  3. 

With  respect  to  the  finding  that  program  complexity 
measures  and  the  number  of  program  interfaces  contributed 
significantly  to  the  predictability  of  error  rate,  studies  by 
Okimoto  and  Thayer  (13,  14),  are  of  interest.  Okimoto  found  in 
a survey  of  approximately  60  systems  programmers  that  when  they 
were  asked  to  rank  order  the  ten  most  important  factors  con- 
tributing to  arror-proneness  in  programs,  poorly  defined  inter- 
faces was  ranked  as  the  highest  and  most  important  factor. 

Other  factors  of  importance  that  were  included  in  this  list  of 
contributors  to  error-proneness  were  (1)  poor/incomplete  testing 
(ranked  3rd) , (2)  complex  function/logic  (ranked  8th) , and 
(3)  large  modules  (ranked  10th). 

In  the  Thayer  et  al . study,  among  the  many  things  that 
were  presented  was  a brief  analysis  of  factors  which  contributed 
to  the  difficulty  of  developing  over  200  command  and  control 
programs.  Each  program  was  rated  according  to  five  categories 
of  difficulty:  difficulty  to  design,  code,  implement,  checkout, 

and  document.  These  ratings  were  then  summed  to  obtain  the 
overall  difficulty  rating  for  each  program.  Of  particular 
interest  were  the  major  reasons  given  for  the  difficult  to 
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develop  programs:  complex  logic,  core  loading  problems,  and 

data  interfaces.  A fourth  reason,  changes  in  interpretation 
of  poorly  stated  requirements,  which  adds  to  complexity  and 
difficulty  in  program  development  was  also  given. 


j 
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6.3  Sample  S Results 
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6.3.1  Errors/Program  = f (Unnormal ized  Variables) 

The  prediction  equation  results  for  both  five  and  ten 
predictor  variables,  when  predicting  errors/program  from  a 
linear  combination  of  the  unnormalized  variables  for  each 
project  of  sample  S,  are  presented  in  Tables  6-1  thru  6-5. 
These  tables  report 

(1)  the  standard  partial  regression  coefficients 

(i.e.,  the  beta  coefficients,  not  the  raw  regression 
coefficients)  for  each  predictor, 

(2)  the  correlation  of  each  predictor  with  errors, 

2 

(3)  the  highest  value  of  R and  R obtained  for  the 
maximum  number  of  predictors  entered  in  the 
equation  by  the  regression  procedure,  and 

(4)  the  analysis  of  variance  tables  for  both  five 
and  ten  predictor  regression  equations. 

Prediction  equation  results  for  projects  f!  and  B where  zero 
errors  were  deleted  from  the  analysis  are  reported  in 
Tables  6-2  and  6-4,  respectively.  Additionally,  Tables  6-6 
and  6-7  are  included  here  to  summarize  the  prediction  results 
obtained  for  both  five  and  ten  predictors  over  all  regression 
equations  that  were  developed  using  the  unnormalized  variables 
as  predictors  of  errors.  For  projects  M,  B,  and  P,  a total  of 
45,  43  , and  45  predictor  variable;;,  rvspec lively,  won;  available 


6-3 


: f- 


I 


j 


TABLE  6-1.  PROJECT  M, 

ERRORS/PROGRAM  = f (UNNORMALI ZED  VARIABLES) 


VARIABLES  IN  PRED 

ICTION 

EOUATI ON 

i 

11 

25CMAX.1 

MULTIPLE  r. 

.717 

.768 

.79/ 

multiple  rz 

.514 

.589 

.636 

STD.  ERROR  OF  ESTIMATE 

7.449 

6.892 

6.621 

VAR  I AOLES  CX) 

COEFFICIENTS 

>1 

X 

-< 

XI 

SOURCE  INSTRUCTIONS 

-1.280 

.563 

X2 

ENTRY  POINTS 

* .17  5 

- .146 

.150 

X4 

USING  INSTRUCTIONS 

.191 

.290 

.507 

X9 

CALLS/LINKS 

.105 

.374 

XI  2 

EQUATE  STATEMENTS 

- .181 

- .257 

. 362 

XI 4 

LOGICAL  CONNECTORS 

— 

- .151 

. 268 

XI  5 

CONDITIONAL  JUMPS 

— 

.605 

.593 

XI 6 

FUNCTIONS 

— 

.182 

.515 

X20 

LOCK  MACROS 

.106 

. 

.309 

X37 

UNDEFINED  VARIABLES 

.734 

.712 

. 6 4 5 1! 

:<54 

SI  X AVG.  NO.  OPERATORS/ 

- - ~ 

.601 

.590 

ARITHMETIC  INSTR. 

ANALYSIS  OP  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

P 

5 

REGRESSION 

22794.547 

5 

4558.906 

82.153 

<.001 

RES  I OUAL 

21535.109 

3R0 

55 .489 

10 

REGRESSION 

26140.555 

10 

2614.055 

55.035 

< .001 

RESIDUAL 

13  239  . 1(12 

384 

47.497 

TOTAL 

44379.660 

394 

" BEST  SINGLE  PREOICTOR 
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T ABLE  6-2.  PROJECT  M, 

ERRORS /PROGRAM  * fQUNNORMALI ZED  VARIABLES), 
ZERO  ERRORS  DELETED 


VARIABLES  IN  PREDICTION  EQUATION 


i II  24CMAX.) 


MULTIPLE  R, 

.69  5 

.755 

. 791 

MULTIPLE  R 

.485 

.571 

.625 

STD.  ERROR  OF  ESTIMATE 

8.215 

7.552 

7.228 

VARIABLES  (X)  j 

COEFFICIENTS 

rXY 

XI 

SOURCE  INSTRUCTIONS 

_ — — 

- .929 

.534 

X2 

ENTRY  POINTS 

- .192 

- .148 

.115 

X4 

USING  INSTRUCTIONS 

.174 

.283 

.456 

X12 

EQUATE  STATEMENTS 

- .183 

- .227 

.355 

X1J 

COMMENTED  instructions 

--- 

.139 

.353 

Xl  5 

CONDITIONAL  JUMPS 

— 

.659 

.584 

X16 

FUNCTIONS 

.238 

.465 

X20 

LOCK  MACROS 

. 104 

- -- 

.290 

X 3 7 

UNDEFINED  VARIABLES 

. 738 

.647 

.624“ 

X42 

NON-NESTED  DO  LOOPS 

— 

.158 

.452 

X53 

INSTR.,  6 TH  LEVEL  OR  LOWER, 

- - - 

.102 

.147 

DO  LOOPS 

ANALYSIS  OF 

VARIANCE 

NO. 

SUM  OF 

MEAN 

PREDICTORS  SQUARES 

OF 

SQUARE  F 

5 REGRESSION 

RESIDUAL 

ID  REGRESSION 

RESIDUAL 

TOTAL 


18866 

.531 

5 

377 

20175. 

,902 

299 

6 

22274. 

,613 

10 

222 

16767, 

.816 

294 

5 

39042 , 

,434 

304 

3.306 

7.473 

55.919 

<.001 

7.461 
7 .vii 

39  .0  5 5 

<.001 

“BEST  SINGLE  PREDICTOR 
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TABLE  6-3.  PROJECT  B, 

ERRORS /PROGRAM  = f ( UNNORMAL  I ZED  VARIABLES) 


VARIABLES  IN  PREDICTION  EQUATION 


1 

JUL 

19CMAX.) 

MULTIPLE  r. 

.312 

.853 

. 886 

MULTIPLE  R ‘ 

.66(1 

.727 

.785 

STD.  ERROR  OF  ESTIMATE 

3.615 

3.322 

3.113 

VARIABLES  (X) 

COEFFICIENTS 

rXY 

X4 

USING  INSTRUCTIONS 

... 

. 245 

.436 

X3 

UNCONDITIONAL  JUMPS 

— 

. 172 

. 294 

X15 

CONDITIONAL  JUMPS 

- .44  3 

- .557 

.582 

X17 

SCALING/ROUNDING  OPNS . 

.211 

.219 

.589 

X22 

ADDRESS  VARIABLES 

— 

. 239 

.319 

X 2 8 

FIXED  POINT  VARIABLES  FREQ. 

— 

. 197 

.498 

X 3 5 

REGISTER  VARIABLES 

- - - 

- .294 

. 4,19 

X 3 7 

UNDEFINED  VARIABLES 

.631 

. 588 

. 67  3" 

X49 

INSTR.,  2ND  LEVEL  00  LOOPS 

. 335 

.313 

.663 

X53 

INSTR.,  6TH  LEVEL  OR  LOWER 

. 286 

. 2 J;i 

.565 

DO  LOOPS 

ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

' 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

P 

5 

REGRESS  1 ON 

2433.154 

5 

496.631 

38.313 

< . J J 1 

RESIDUAL 

1283  , 795 

9 S 

13.369 

1,1 

REGRESSION 

2 7 3 7.6  1.! 

13 

273.761 

2 4.836 

< . .!  3 1 

RESIDUAL 

U27.339 

9 3 

11.’  36 

TOTAL 

3763.949 

M3 

"•  BEST  SINGLE  PREDICTOR 
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TABLE  5-4.  PROJECT  B, 

ERRORS/PROGRAM  = f (UNNORMAL I ZED  VARIABLES), 
ZERO  ERRORS  DELETED 


VARIABLES  IN 

PREDICT  I ON 

EOUATI ON 

i 

i£ 

14CMAX.) 

MULTIPLE  R, 

. 8J4 

.895 

.919 

MULTIPLE  R 

.595 

.79  7 

. 845 

STD.  ERROR  OF  ESTIMATE 

3.418 

2.869 

2,572 

VARIABLES  (X) 

COEFFICIENTS 

CXY 

X4 

USING  INSTRUCTIONS 

» — — 

. 2911 

.658 

X6 

LABELED  INSTRUCTIONS 

- - - 

.275 

.497 

XU 

USER  MACROS 

— 

.431 

.4111 

X15 

CONDITIONAL  JUMPS 

- .493 

- .335 

.573 

X18 

SHORT  DO  LOOPS 

— 

.194 

.IJ2 

X27 

FIXED  POINT  VARIABLES 

.154 

.286 

.333 

X37 

UNDEFINED  VARIABLES 

.695 

.71)  l) 

.635 

X49 

INSTR.,  2ND  LEVEL  00  LOOPS 

.35J 

.482 

. 688” 

X53 

1NSTR.,  6 TH  LEVEL  OR  LONER 
DO  LOOPS 

.359 

.474 

.590 

X54 

SI  X AVG.  NO.  OPERATORS/ 
ARITHMETIC  INSTR. 

-1.056 

.604 

ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

F 

5 

REGRESSION 

RESIDUAL 

2571.739 

1339.876 

39 

U 74  . 3 U B 

11,683 

40.602 

<.001 

111 

REGRESSION 
RES  1 DUAL 

2 720 . 190 
691 .336 

13 

84 

172.JI9 

8,230 

33.051 

<.001 

TOTAL 

JUlt .526 

94 

“BEST  SINGLE  PREDICTOR 
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TABLE  6-5. 


PROJECT  P, 
ERRORS/PROGRAM 


= f (UNNORMALI ZED  VARIABLES) 


tfLES 


VARIABLES  IN  PREDICTION  EQUATION 


i lit  17CMAX,) 


MULTIPLE  R, 

MULTIPLE  R^ 

STD.  ERROR  OF  ESTIMATE 


~.R§8  . 9 46  .988 

. 7 a'B"  . 89  5 .9  7 7 

2.6  1 3 2 .(3  22  1 . 1 3(1 


VARIABLES  (X) 


COEFFICIENTS  rxy 


X4 

USING  INSTRUCTIONS 

— 

- .465 

.552 

X5 

COMMENT  STATEMENTS 

.317 

. 762 

.539 

X8 

UNCONDITIONAL  JUMPS 

.74  3 

. 644 

.664“ 

X 1 7 

SCALING /ROUNDING  OPMS , 

- - - 

- .176 

- . J 2 4 

X 2 3 

ADDRESS  VARIABLE  FREQ. 

.57*3 

- . JS6 

.168 

X28 

FIXED  POINT  VARIABLE  FREQ. 

.324 

.60-1 

. 242 

X37 

UNDEFINED  VARIABLES 

- - - 

.436 

.56  7 

X42 

NON-NESTED  DO  LOOPS 

- — » 

- .386 

.179 

X47 

DO  LOOPS,  6 TH  LEVEL  OR  LOWER 

.336 

— 

.345 

X51 

1NSTR.,  4 TH  LEVEL  00  LOOPS 

- - - 

- .374 

.424 

XS5 

INSTR.,  6 TH  LEVEL  OR  LOWER 

— 

.632 

.341 

DO  LOOPS 


ANALYSIS  OF  VARIANCE 


NO  . 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

V' 

5 

REGRESSION 

737.633 

5 

147.527 

21.612 

<.  3 i 1 

RES  I DUAL 

197.462 

29 

6.826 

111 

REGRESSION 

837.431 

l.’ 

35.743 

2 if  . 474 

<. .! ,!  1 

RES IDUAl 

98.165 

24 

4 . 3 9 0 

TOTAL 

955.596 

54 

:l  BEST  SINGLE  PREDICTOR 
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TABLE  6-7.  TEN  PREDICTOR  SUMMARY, 

errors /program  = ecumnormal i zed  variables) 


VARIABLES 

Ma 

Mb 

Ba 

Bb 

P 

REGRESSION  COEFFICIENTS 

X4 

USING  INSTRUCTIONS 

. 29(1 

.283 

. 245 

.293  - 

.465 

X57 

UNDEFINED  VARIABLES 

. 712“ 

. 647“ 

. 588“ 

.733 

.436 

XI  5 

CONDITIONAL  JUMPS 

.633 

,659  - 

.557 

- .335 

X 53 

INSTR,,  6TH  LEVEL  DO  LOOPS 

.10  2 

. 2 3 if 

.4  74 

.632 

XI 

SOURCE  INSTRUCTIONS  -1 

.2  83  - 

.929 

X 2 

ENTRY  POINTS 

. 1 4 ft  - 

.148 

X3 

UNCONDITIONAL  JUMPS 

.172 

. 544“ 

X 1 2 

EQUATE  STATEMENTS 

. 2 3 7 - 

.227 

X 1 7 

SCALING/ROUNDING  OPUS . 

.219 

- 

.176 

X2  3 

FIXED  PT,  VAR.  FREQ. 

. 197 

.63:3 

X 42 

NON-NESTED  DO  LOOPS 

.152 

- 

. 336 

X49 

INSTR.,  2ND  LEVEL  DO  LOOPS 

.313 

. 474“ 

X 54 

SI  X AVG.  NO.  OPERATORS/ 

.611 

-1.356 

ARITHMETIC  INSTR. 

CORRELATION  STATISTICS 


rSI , ERRORS 

.563 

,534 

.649 

.653 

.593 

r " , S 1 

. 88:1 

.872 

.359 

.7  27 

.5  37 

r“, ERRORS 
r “, ERRORS 

.645 

.624 

.673 

. 6 S3 

. 66  4 

.4  11', 

. 339 

. 4C  3 

.473 

.44  1 

PREDICTION  SUMMARY 

R-i 

. 76  6 

.755 

.35  3 

.413 

. 946 

Rl 

. :>  •••  j 

.57  1 

. 7 V 

. 7)  7 

. 39  5 

aALL  OBSERVATIONS  USED 
bHERO  ERRORS  DELETED 

Brest  single  PRFntrTnp 
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to  be  automatically  entered  in  the  prediction  equation  (see 
Table  C-l,  Appendix  C) . 

Initial  observation  of  these  results,  particularly  the 

2 

R values  as  reported  in  Tables  6-6  and  6-7,  indicates  that 

(1)  for  all  sample  S projects  the  increase  in  R ; i.e.,  the 

percent  of  variance  accounted  for  when  predicting  errors, 

shows  a moderate  increase  when  ten  predictors  are  used  as 

opposed  to  when  only  five  are  used;  and  (2)  the  predictability 

2 

of  errors  ranges  from  moderate  for  project  M (R  ■ .589)  to 

2 

high  for  project  P (R  - .895).  One  can  also  observe  (see 
Tables  6-1  thru  6-5)  that  the  variables  selected  for  the  five 
predictor  solution  are  predictors  which  generally  appear  again 
in  the  ten  predictor  solution,  after  five  different  predictors 
have  been  entered. 

Clearly  the  predictability  of  errors  in  sample  S is  not 
consistently  high  over  each  project.  Two  predictors  however, 

Using  Instructions  and  Undefined  Variables,  do  appear  con- 
sistently in  each  of  the  ten  predictor  equations  ior  each 
project.  Other  variables  as  indicated  in  Table  6-7  do  appear 
less  consistently  either  for  two  or  more  of  the  prediction 
equations  that  were  developed,  or  for  at  least  two  ot  the  pro- 
jects of  sample  S.  The  fact  that  variables  may  appear  con- 
sistently over  two  or  more  projects,  however,  does  not  necessarily 
mean  that  they  are  the  most  important  variables  from  a prediction 
viewpoint . 

The  best  single  predictor  of  errors  in  each  project  is 
evaluated  by  its  correlation  with  errors  and  not  its  beta 
coefficient  in  the  regression  equations.  For  projects  M and 
B,  Undefined  Variables  was  the  best  single  predictor  of  errors. 
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For  project  P the  beat  single  predictor  was  the  number  of 

Unconditional  Jumps  in  the  program.  However,  in  Table  6-7 

when  one  examines  the  percent  of  variance  in  errors  explained 

2 

by  the  best  single  predictor  (designated  as  r+ ,err0r3 ) 
versus  the  percent  of  variance  explained  by  the  best  single 
predictor  plus  other  program  complexity  variables  (i.e., 

R ) , it  is  observed  that  the  other  program  complexity  variables 
selected  by  the  regression  procedure  are  significantly  con- 
tributing to  the  predictability  of  errors  over  and  above  that 
which  can  be  predicted  only  by  the  best  single  predictor  in 
any  of  the  projects. 

One  additional  point  is  important  enough  to  mention 
here.  In  Table  6-7  the  correlation  statistics  are  reported 
for  the  correlations  between  the  best  single  predictors  and 
Source  Instructions  (SI)  . In  general,  all  of  these  correla- 
tion coefficients  are  moderate  to  high  with  the  exception  of 
the  correlation  of  project  P's  best  single  predictor,  Uncon- 
ditional Jumps,  with  SI.  This  clearly  shows  that  each  best 
single  predictor  is  itself  reflecting  length  of  program  plus 
some  contribution  to  the  prediction  of  errors  which  is  unique  to 
the  variable  itself.  In  essence  then,  given  that  these  best 
single  predictors  had  not  been  used,  source  instructions  by 
itself  could  be  used  to  predict  errors  almost  as  well  as  each 
best  single  predictor. 

With  respect  to  error  predictability,  whet  zero  errors 
are  left  in  or  taken  out  of  the  data  samples  (see  Tables  6-1 
thru  6-u) , predictability  of  errors  increases  appreciably 
(from  .73  to  .80)  for  project  B,  but  no  change  of  any 
statistical  significance  (a  decrease  from  .59  to  .57) 
occurs  for  project  M.  Although  only  nine  zero  error  observations 
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were  deleted  from  project  B's  data  sample,  this  deletion  was 
sufficient  to  cause  changes  in  the  correlations  among  pre- 
dictors and  between  each  of  the  predictors  and  errors,  such 
that  an  almost  completely  different  prediction  equation  was 
produced,  wherein  only  three  of  the  ten  original  predictors 
were  reselected.  For  project  M,  90  zero  error  observations 
were  deleted  and  seven  out  of  ten  original  predictors  in  the 
regression  equation  using  all  observations  (i.e.,  395)  stayed 
in  the  equation*  however,  with  slightly  different  coeffients. 

Several  explanations  for  these  results  can  be  suggested 
at  this  time.  First,  as  earlier  hypothesized  in  Section  3.0, 
programs  with  zero  reported  errors  are  programs  that  have  not 
been  thoroughly  tested  and  contain  latent  errors . Zero  reported 
errors  in  these  cases  are  actually  under-estimates  of  the 
total  errors  in  the  program,  and  if  these  programs  are  used 
along  with  programs  that  have  errors  reported  in  them  for  error 

prediction  purposes,  than  estimates  of  error  predictability; 

2 

i.e.,  values  of  R , will  be  biased  downward.  This  is  in  fact 
what  was  observed  for  project  B. 

2 

On  the  other  hand,  for  project  M,  since  the  change  in  R 
waB  so  slight  (i.e.,  - .02)  and  since  seven  of  the  ten  original 
predictors  remained  in  the  equation,  this  suggests  the  possi- 
bility that  the  90  zero  error  programs  that  were  deleted  might 
have  been  the  shorter  programs  which  had  only  a limited  amount 
of  variability  in  their  predictor  variables.  In  other  words, 
predictor  variables  wuich  may  have  limited  variability  to  con- 
tribute to  the  prediction  of  errors,  when  eliminated  from  the 
analysis,  do  not  significantly  change  or  affect  the  results. 
However,  a validation  of  thi3  suggested  hypothesis  for  the  90 
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error  free  programs  of  project  M can  only  be  attempted  by 
a complete  analysis  of  these  programs,  which  was  not  the 
intent  or  purpose  of  this  study. 
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Interpreting  the  regression  equations  presented  in 
Tables  6-1  to  6-5  presents  some  difficulty.  There  are  several 
reasons  for  this.  One  basic  reason  is  that  beyond  the  descrip- 
tion of  each  variable  that  is  provided  in  Table  B-l,  there  is 
no  more  definitive  understanding  of  what  each  variable  is 
measuring,  not  to  mention  how  combinations  of  these  variables 
should  conceptually  interact  to  influence  errors  in  programs. 

One  predictor  variable  that  would  stand  as  an  exception  for 
which  an  interpretation  could  be  given  in  this  caBe  is 
X5,  Comment  Statements.  Comment  Statements  appears  in  Table  6-5 
as  a predictor  with  positive  coefficient  for  both  the  five  and 
ten  predictor  regression  equations  for  project  P.  One  would 
initially  expect  that  since  comment  statements  are  not  execut- 
able, they  cannot  contribute  to  errors  in  programs.  Thera 
is  no  disagreement  with  this  explanation.  However,  one 
possible  explanation  for  this  variable  appearing  in  the  equa- 
tions for  project  P could  be  that  these  programs  may  have  had 
a comparatively  larger  number  of  comment  statements  in  them 
relative  to  programs  of  project  M and  B.  As  such,  this  in- 
creased the  readability  of  the  programs,  which  resulted  in 
more  errors  being  found  because  they  were  more  easily  detected. 

An  additional  reason  why  no  straight-forward  interpretation 
of  the  regression  equations  can  be  attempted  is  that  each  of 
the  predictors  that  were  selected  were  usually  found  to  be 
correlated  with  other  variables  in  the  equation.  This  was 
particularly  evident  for  the  results  presented  in  Table  6-1 
for  project  M.  For  example,  one  might  ask  the  reason  for  the 
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negative  beta  coefficients  for  variable  Xl,  Source  Instruc- 
tions, when  it  is  reported  as  being  positively  correlated  with 
errors.  Statistically,  there  is  an  explanation  for  the  large 
negative  weight  being  computed  for  Source  Instructions.  Other 
predictors,  particularly  those  that  had  entered  the  equation 
prior  to  Source  Instructions,  were  moderately  to  highly  corre- 
lated with  Source  Instructions.  The  correlation  coefficients 
of  each  of  the  predictors  with  Source  Instructions  is  reported 
below: 


X2  X4  X9  X12  X14  X1S  X16  X37  X54 

.352  .535  .627  .633  .425  .910  .646  .880  .976 


After  the  first  variable  had  entered  the  equation  (i.e., 
variable  X37 , the  best  single  predictor  of  errors),  the  partial 
correlation  of  Source  Instructions  with  errors  was  very  close  to 
zero.  As  such,  when  Source  Instructions  entered  the  equation 

which  was  at  step  7 in  the  regression  procedure,  it  contributed 

n 

to  the  predictability  of  errors  (i.e.,  R“)  by  suppressing  the 
effect  of  Source  Instructions  from  other  variables  that  had 
already  been  entered  in  the  equation.  This  then  i3  the  rea- 
son for  the  large  negative  weight  being  computed  for  Source 
Instructions.  By  suppressing  or  taking  out  this  effect,  the 
predictive  p<Mojr  of  the  other'  v.ir  i .il>l  e;:  w.is  i ncr<-.i::n<| , . i r.  wa;; 
evidenced  by  positive  changes  in  these  predictor's  coefficients. 
In  essence  then  variable  XI  is  functioning  as  a suppressor 
variable  in  the  equation  of  Table  6-1.  Suppressor  variables 
are  quite  common  in  the  social  sciences  and  in  other  areas  of 
study  wherein  measurement  tools  are  either  unavailable  or  have 
not  been  developed  for  providing  unique  and  independent 
measures  of  predictor  variables  for  data  collection  and  analysis 
purposes.  (Suppressor  variables  generally  appeared  throughout 
the  results  for  both  samples  S and  T) . For  a more  detailod 
discussion  of  suppressor  variables  the  reader  should  consult 
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the  following  references  (McNemar,  1962;  Van  de  Geer,  1971; 

Harris,  1975)  (6,  10,  16) . 

Finally,  when  there  is  a moderate  or  greater  degree  of 
multicollinearity  among  the  predictors  in  the  regression  equation, 
making  an  assessment  of  the  relative  importance  of  the  indepen- 
dent variables  for  predicting  errors  is  not  readily  accomplished 
by  inspecting  only  the  beta  coefficient  for  a given  predictor 

(Ferguson,  p.  402)  (4).  For  example,  with  two  predictors  the 
2 

value  of  R can  be  shown  to  be  equal  to 

R2  - Bx2  + Bj  + 2B1B2r12  . ' (6.0) 

In  this  case  the  predicted  variance  is  comprised  of  three  additive 
parts.  B1  represents  a contribution  by  predictor  X^,  B2  a con- 
tribution by  X2,  and  2B1B2r12  is  a component  which  involves  the 
correlation  between  X^  and  X2 . Clearly  then,  evaluating  the 
relative  contribution  of  a predictor  in  the  multiple  regression 
equation  requires  that  correlation  terms  and  other  predictor's 
beta  coefficients  be  considered  simultaneously.  This  would  not 
be  the  case,  however,  had  each  of  the  predictors  been  statis- 
tically independent  of  other  variables  in  the  equation. 

6.3.2  Errors/Program  - f(SI  + Normalized  Variables) 

As  previously  discussed  in  Section  5.0,  prediction  equations 
for  errors/program  were  also  developed  using  the  normalized 
variables  as  predictors  in  combination  with  the  Source  Instruc- 
tions (SI)  variable,  XI.  The  results  obtained  from  these 
predictions  provide  information  on  how  effectively  the  origi- 
nal variables,  once  the  effect  of  Source  Instructions  had  been 
"removed''  from  each  predictor,  combined  with  XI  to  predict 
errors.  The  prediction  equation  results  are  presented  in 
Table  6-8  through  6-12.  Summary  results  for  five  and  ten 
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TABLE  6-3. 


PROJECT  M, 

ERRORS /PROGRAM  = f C S I + NORMALIZED  VARIABLES) 


VARIABLES  IN  PREDICTION  EQUATION 


i 

il 

18CMAX.) 

MULTIPLE  R, 

.627 

.691 

.654 

MULTIPLE  r 

. 3 8 7 

.911 

.928 

STD.  ERROR  OF  ESTIMATE 

8.562 

8.251 

8.223 

VARIABLES  (X) 

COEFFICIENTS 

c 

xy 

XI 

SOURCE  INSTRUCTIONS 

.597 

.569 

. 56  3s* 

X 5 7 

EXIT  POINTS/SI 

— 

- .,<9  3 

- .213 

X 5 8 

USING  INSTRUCTIONS /SI 

. I 5J 

.131 

- .126 

X69 

CONDITIONAL  JUMPS/SI 

. 193 

. 12J 

.132 

X7J 

FUNCTIONS/S  1 

.195 

.155 

.232 

X71 

S CAL  I NG.' ROUND  I MG  OPNS./Si 

- - - 

.3  70 

.16  3 

X74 

LOCK  MACROS/SI 

.19  3 

.391 

.MOV 

X8Z 

FIXED  PT.  VAR.  FREQ/SI 

--- 

- .3  65 

- .39,1 

X9  J 

REGISTER  VAR.  FREQ./S I 

— 

- « 

- .171 

XI J 8 

SI  X AVG.  NO.  OPERATORS/ 

— 

. :!  6 7 

.116 

ARITHMETIC  IN3TR./SI 

ANALYSIS  OF 

VARIANCE 

NC. 

PREDICTORS 

SUM  OF 
SQUARES 

OF 

MEAN 

SQUARE 

r 

P 

5 

REGRESSION 

17178.586 

5 

3935.717 

49. 

! 34  <.001 

RES IDUAL 

27201.070 

389 

69.926 

10 

REGRESSION 

18234,863 

10 

1823.486 

26  . 

7 3 2 < . il  .1 1 

RESIDUAL 

2 6 1 •*  9 , 7 9 3 

339 

68.3  3 5 

TOTAL 

94379 .66  < 

39  9 

" BEST  SINGLE  PREDICTOR 
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TABLE  6-9.  PROJECT  M, 

F.RRORS/PROGRAM  s f(S]  + NORMALIZED  VARIABLES), 
ZERO  ERRORS  DELETED 


VARIABLES  IN  PREDICTION  EQUATION 


J>  H 1 UMAX.  ) 


MULTIPLE  R, 

MULTIPLE  R‘ 

STD.  ERROR  OF  ESTIMATE 

.598 
.557 
9 . 161 

.617 

.381 

9.069 

.619 

.383 

9.365 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

XI 

SOURCE  INSTRUCTIONS 

.559 

.599 

. 5 3 3“ 

X57 

EXIT  POINTS/SI 

--- 

- .J83 

- .195 

xsa 

USING  INSTRUCTIONS/S I 

.156 

. 14R 

- .16,1 

X6l 

ARITHMETIC  I NS  TRUCT  I Of.S /S  I 

— 

- .0  66 

- .075 

X69 

CONDITIONAL  JUMPS /SI 

.216 

.16  0 

.2  46 

X7,1 

FUNCTIONS/S  I 

.169 

.147 

.142 

X74 

LOCK  MACROS /SI 

.'19  4 

.076 

.0  66 

x 37 

LABELED  ARRAY  VARIAQLE3/SI 

... 

- . 164 

- .003 

X92 

UNDEFINED  VAR.  FREQ. /SI 

... 

.■'9  4 

.113 

xua 

SI  X AVG.  NO.  OPERATORS/ 
ARITHMETIC  INSTR./St 

.089 

.162 

ANALYSIS  OF  VARIANCE 


NO  . 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

1- 

5 

REGRESSION 

RESIDUAL 

13949.352 

25092.573 

5 

299 

2739.973 

33.922 

33.245 

<.001 

10 

REGRESSION 
RES  I DUAL 

14362.015 

24179,816 

10 

294 

1436  , 26  1 
32 . 244 

13.071 

<.001 

TOTAL 

59042.434 

50  4 

i!  BEST  SINGLE  PREDICTOR 
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TABLE  6-10.  PROJECT  B, 

ERRORS/PROGRAM  s £(S1  + NORMALIZED  VARIABLES) 


VARIABLES  I N PREDICTION  EQUATION 


i 

11 

1 4 c MA  X . ) 

MULTIPLE  R 

.71? 

.754 

.772 

MULTIPLE  R2 

.51) 

.564 

. 596 

STD.  ERROR  OF  ESTIMATE 

4.32) 

4.178 

4.1)6 

VARIABLES  (X) 

COEFFICIENTS 

>i 

X 

Vi 

XI 

SOURCE  INSTRUCTIONS 

.587 

.651 

.6491! 

X57 

EXIT  POINTS/SI 

— 

. 169 

- .057 

X6  2 

UNCONDITIONAL  JUMPS/SI 

— 

.110 

- .090 

X 7 1 

SC AL I NG / ROUND  1 NG  OPNS./SI 

, • .256 

, 344 

X32 

FIXED  PT.  VAR.  FREO./St 

. 104 

— 

.358 

X3  3 

FLOATING  PT.  VARIABLES/SI 

— 

- .16,1 

- .06  5 

X39 

REGISTER  variables/si 

— 

- .114 

- .316 

X91 

UNDEFINED  VARIABLES /SI 

— 

.269 

- .273 

X99 

DO  LOOPS,  NESTED  AT  4Th 
LEVEL/SI 

- .446 

- .29) 

.14) 

XI J 3 

INSTR.,  2ND  LEVEL  DO 
LOOPS/SI 

. 193 

.243 

.241 

XI 0 5 

INSTR.,  4TH  LEVEL  DO 
LOOPS /SI 

.433 

.422 

.210 

ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

5 

REGRESS ! ON 

19  3 2.  1 41, 

5 

386.429 

2 .1  . 6 7 4 <• 

RES  1 DUAL 

1 3 3 1 . 8 .1 4 

9 8 

13.692 

10 

REGRESSION 

2140.797 

1,1 

214.080 

12.266  < 

RES  I DUAL 

1623.151 

93 

1 7 . 4 5 3 

TOTAL 

3763.949 

1 ! 3 

*B 

* * ' * 


s!  JtST  S INGLE  PKliOtCTOK 
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TABLE  6-11. 


PROJECT  B, 

ERRORS /PROGRAM  s £ ( S I + NORMALIZED  VARIABLES) 
ZERO  ERRORS  DELETED 


VARIABLES  IN  PREDICTION  EQUATION 


1 

i! 

1 EC  MAX  , ) 

MULTIPLE  R, 

.7)5 

.777 

.787 

MULTIPLE  rz 

. 541 

.604 

.619 

STD.  ERROR  OP  ESTIMATE 

A.  196 

4. ,109 

5.982 

VARIABLES  CX) 

COEFFICIENTS 

T 

4 xy 

XI 

SOURCE  INSTRUCTIONS 

. 6 4b 

.608 

. 6 5 8!! 

X 5 8 

USING  INSTRUCTIONS/SI 

- -- 

.238 

- .230 

X59 

COMMENT  STATEMENTS /S 1 

— 

- .264 

- .3  35 

X6  5 

USER  MACROS /St 

— 

.131 

. ;i  5 1 

X76 

ADDRESS  VAR  I AULES/S I 

- -- 

. 103 

.137 

X8  2 

FIXED  PT.  VAR.  FREQ ./Si 

— 

.122 

. 366 

X9  1 

UNDEFINED  VARIABLES/S  1 

.15  2 

. 248 

- .29,! 

XI  33 

INSTR,,  2ND  LEVEL  00 
LOOPS/S  1 

.253 

. 3(16 

.26  5 

X 1 J 7 

INSTR.,  6 TH  LEVEL  OR  LOWER 
DO  LOOPS/SI 

, 2 '<6 

.216 

, 4J3 

X 1 iJ  8 

SI  X AVG.  NO.  OPERATORS/ 
ARITHMETIC  INSTR, / 5 I 

- .137 

- .134 

- .061 

ANALYSIS  OF  VARIANCE 


NO. 

PRED 1 CTCRS 

SUM  OF 
SQUARES 

UF 

MEAN 

SQUARE 

5 

REGRESSION 

1344,559 

5 

348.912 

2 i'  .953  < . .J  ,1 1 

RES  I DUAL 

1566.967 

33 

17  .6,16 

IJ 

REGRESSION 

2,16  1 .177 

1 1 

2:16  , 113 

1 2 . i’  2 2 < . ,1 .)  1 

RES  1 DUAL 

135J.349 

34 

lb  .J7b 

TOTAL 

5411,526 

94 

11  BEST  SINGLE  PREDICTOR 


6-25 


•i  *4 


TABLE  6-12. 


PROJECT  P, 

ERRORS /PRO ARAM  = f(SI  + NORMAL ! ZED  VARIABLES) 


VARIABLES  IN  PREDICTION  EQUATION 


i 

19 

13CMAX.9 

MULTIPLE  r. 

. BBS 

.934 

■ 9 36 

MULTIPLE  R 

.774 

.871 

.972 

s. 

STD.  ERROR  OP  ESTIMATE 

2.698 

2.239 

1.276  i 

■ 

VARIABLES  Cx) 

COEFF I C 1 ENTS 

rxy  j 

f XI 

SOURCE  INSTRUCTIONS 

.682 

. 334 

.1 

. 5 9 3;t  / 

i X59 

COMMENT  STATEMENTS /SI 

. 26  1 

.259 

.235  ; 

i:  x6  2 

UNCONDITIONAL  JUMPS/SI 

.4/9 

.612 

.543 

i X63 

CALLS /L I NXS/SI 

— 

.198 

- ,352 

1 X69 

CONDITIONAL  JUMFS/S 1 

. 196 

— 

.2  24  1 

’ X71 

SCALING/ROUND  I NO  OPNS./SI 



- .26  1 

- .49  6 

j X7  7 

ADDRESS  VAR.  FREQ. /SI 

— 

- .135 

- .3  22 

. X3  2 

FIXED  PT.  VAR.  FRED. /SI 

- -- 

.369 

-.9  53  ’• 

X0  6 

NON-NES  TED  DO  LOOPS/SI 

— 

- .3)2 

- .393  1 

1 X9  7 

00  LOOPS  NESTED  AT  2ND 

- -- 

. 167 

.147  ■! 

LEVEL/SI 

' X 10  2 

INSTR.  in  non-nested 

- .216 

... 

-.  3 5 5 .s 

i ■ 

00  LOOPS /SI 

!j  Xll)  7 

INSTR,  IN  6 TH  LEVEL  OR 

— 

. 242 

.2  24  l 

LOWER  DO  LOOPS/SI 

, 

[ 

ANALYSIS  OF 

VARIANCE 

1 NO. 

SUM  OF 

MEAN 

| PREDICTORS  SQUARES 

OF 

SQUARE 

F * 

5 

REGRESSION  724.430 

5 

144 . 39  7 

19.934  <-.  J J 1 

RESIDUAL  211. 1 1J 

2 9 

7 . 230 

1.1 

REGRESSION  315.275 

1.J 

81.528 

lh.l’I.J  <.9  31 

RESIDUAL  12J.J21 

2 4 

5 . 1)  1 3 

TOTAL  9 J 5.  5 O', 

34 

— 

11  BEST  SINGLE  PREDICTOR 
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“zero  errors  deleted 

"BEST  SINGLE  PREDICTOR 


TABLE  6-14.  TEN  PREDICTOR  SUMMARY , 

ERRORS /PROGRAM  s £(SI  + NORMALIZED  VARIABLES) 


VARIABLES 

Ma 

Mb 

B* 

Bb 

P 

REGRESSION  COEFFICIENTS 


XI 

SOURCE  INSTRUCTIONS 

. 5 6 4<‘ 

. 549" 

. 6 5 1 !l 

. 60  8!! 

. 334!! 

X57 

EXIT  POINTS/SI 

. 098  - 

.083 

. 169 

X 5 8 

USING  INSTRUCTIONS/5  I 

.181 

.149 

. 208 

X71 

SCALING/ROUNDING  OPNS . /S ! 

.070 

.256 

- .261 

X82 

FIXED  PT.  VAR.  FREO./S!  - 

.063 

.122 

. 369 

XUJ8 

S I X AVG.  NO.  OPERATORS/ 

.06  7 

.0  89 

- .134 

ARITHMETIC  INSTR./SI 

X59 

COMMENT  STATEMENTS /SI 

. 259 

X6  2 

UNCONDITIONAL  JUMPS/SI 

. 1 1? 

- .264 

.612 

X69 

CONDITIONAL  JUMPS /SI 

. 123 

.160 

X72 

FUNCTIONS /SI 

.155 

.147 

.<74 

LOCK  MACROS/S  I 

.091 

.076 

X91 

UNDEFINED  VARIABLES/SI 

. 269 

.248 

X10  3 

INSTR. , 2ND  LEVEL  DC 

. 243 

.306 

LOOPS/S! 

XI  J 7 

INSTR.,  6TH  LEVEL  DO 

.216 

. 224 

LOOPS/S  I 

CORRELATION  STATISTICS 

r>'.  ERRORS 

.563 

.533 

.649 

.658 

.593 

ERRORS 

.317 

. 234 

.421 

.433 

.352 

PREDICTION  SUMMARY 

R2 

.641 

.617 

.754 

.777 

.934 

R2 

.411 

. 381 

.569 

. 6 J 4 

.871 

®ALL  OBSERVATIONS  USED 
bZER0  ERROR  RATES  DELETED 


!IBEST  SINGLE  PREDICTOR 


predictors  are  presented  in  Tables  6-13  and  6-14,  respectively. 

For  projects  M,  B,  and  P a total  of  45,  43,  and  45  predictor 
variables,  respectively,  were  available  to  be  automatically 
entered  in  the  prediction  equation  (see  Table  C-l,  Appendix  C) . 

The  predictability  of  errors  is  improved  only  moderately 

when  ten  versus  five  predictors  are  used  in  the  equation. 

2 

However,  except  for  project  P (R  -.87),  the  predictability  of 

2 

errors  is  generally  low  to  moderate,  ranging  from  R “.38  for 
2 

project  M to  R “.60  for  project  B,  using  a combination  of  XI 
plus  selected  normalized  predictors.  Judging  from  the  summary 
data  provided  in  Tables  6-13  and  6-14,  there  appears  to  be 
little  consistency  of  predictors  appearing  in  the  equations  for 
each  project.  For  five  predictors,  only  one  predictor,  Source 
Instructions,  appeared  in  each  of  the  five  prediction  equations 
that  were  developed  across  all  three  projects,  with  variable 
XR9,  Conditional  Jump^/SI , appearing  only  for  projects  M and  P. 

Tor  ten  predictors,  only  three  variables,  XI,  Source  Instructions, 
X71,  Scaling/Rounding  Opera  l:  ions /f>T , and  X83,  Fixed  Point  Variable 
Frequency/SI , appeared  in  one  or  more  of  the  equations  developed 
for  each  project. 

Source  Instructions  is  the  best  single  predictor  in  each 
of  the  equations,  and  when  considered  by  itself,  as  also  indi- 
cated by  the  results  of  the  previous  section,  it  can  be  used  to 
account  for  a large  percentage  of  the  variation  in  errors  that 
is  accounted  for  by  the  combined  set  of  10  predictors  selected 
for  each  project's  equation (s). 

For  equations  that  were  developed  when  zero  errors  were 

deleted  (Tables  6-9  and  6-11) , it  is  found  for  ten  predictors 

2 

that  only  minor  changes  resulted;  R decreased  .03  for  project  M 
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and  increased  .04  for  project  B,  with  respect  to  the  pre- 
dictability of  errors  using  the  normalized  variables. 

However,  as  observed  in  the  results  of  Section  6.3.1,  the 
variables  in  the  prediction  equations  do  change  when  these 
zero  error  program-observations  are  deleted.  For  project  B 
only  four  of  the  ten  predictors  using  all  observations  were 
reselected  when  nine  zero  error  observations  were  deleted, 
whereas  for  project  M seven  of  the  ten  predictors  remained 
in  the  equation  when  90  observations  were  deleted  from  the 
analysis . 

The  low  predictability  obtained  when  using  the  normalized 
variables  seems  to  result  for  two  reasons.  First,  as  reported 
in  Tables  6-8  through  6-12,  the  correlation  of  each  normalized 
variable  with  errors  was  generally  very  low.  This  indicates 
that  the  variable  errors  per  program  has  very  little  in  common 
with  these  normalized  variables  other  than  Source  Instructions. 

A second  reason  for  this  low  predictability  results  from  the 
fact  that  a majority  of  these  normalized  variables  were 
uncorrelated  with  each  other.  Thus,  each  variable  i3  contri- 
buting independently  to  the  prediction  of  errors,  with  only 
a very  small  chance  that  a suppressor  variable;  i.e.,  a variable 
with  zero  or  low  correlation  with  errors  and  correlated  highly 
with  a predictor  that  correlates  moderately  to  highly  with 
errors,  could  be  present  that  would  improve  the  predictability 
of  these  variables. 

The  results  for  project  P are  of  interest  here,  not  only 

2 

because  of  the  high  predictability  obtained  (R  *,871),  but  also 

because  this  R value  was  only  ,024  less  than  the  value  of 
2 

R (.895)  obtained  when  errors  were  predicted  as  a function  of 

the  unnormalized  variables.  For  each  of  the  other  two  projects, 

2 

the  reductions  in  R values  were  much  larger  (.18  for  project  M 
and  .16  for  project  B) . 
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Comparing  the  predictors  selected  for  project  P in 
Table  6-5  (i.e.,  when  errors  were  predicted  as  a func- 
tion of  the  unnormalized  variables)  versus  the  predictors 
selected  in  Table  6-12  (when  errors  were  predicted  as  a function 
of  Source  Instructions  plus  the  normalized  variables) , one 
observes  that  seven  of  the  ten  unnormalized  predictions  of 
Table  6-5  ware  reselected  in  their  normalized  form  for  the 
equation  in  Table  6-12.  The  results  of  each  table  are  repeated 
in  Table  6-15  for  ease  of  comparison.  (For  project  M with  395 
observations,  only  four  of  the  ten  original  variables  were  re- 
selected  in  their  normalized  form,  and  for  project  B with  10  4 
observations,  five  of  the  ten  were  reselected) . 

In  Table  6-15  each  of  the  correlations  of  unnormalized 
variables  with  errors  changed  in  a negative  direction  when 
the  respective  variable  was  normalized.  For  some  variables 
(X5,  X9,  X23,  X23,  and  X53)  this  meant  that  their  correlations 
became  smaller  in  magnitude,  whereas  for  others  (X17  and  X42) 
their  correlation  with  errors  increased  in  magnitude. 

For  project  P this  high  predictability  of  errors  (R2«.87)  is 
consistent  with  the  high  predictability  (R2=.89)  obtained  when 
errors  are  predicted  using  the  unnormalized  variables  (see 
Section  6.3.1),  and  indicates  that  predictability  is  relatively 
unaffected  when  the  original  program  complexity  variables,  with 
the  effect  of  Source  Instructions  removed,  are  combined  with 
Xl  to  predict  errors.  One  possible  explanation  for  this  con- 
sistency of  prediction  using  different  sets  of  predictors 
could  be  that  project  P implemented  several  new  programming 
techniques  during  its  development,  whereas  projects  M and  B 
did  not  to  any  great  extent.  Another  explanation  could  be 
that  only  a small,  select  group  of  particularly  skilled  pro- 
grammers wore  needed  to  develop  the  35  programs  of  project  P; 
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TABLE  6-15.  PROJECT  P,  PREDICTION  EQUATION  COMPARISON 


RESULTS 

VARIABLES 

COEFFICIENT 

r 

XV 

Table  6-5, 

■xu 

Using  Instructions 

-.465 

.352 

R‘- , 895 

xs 

Comment  Statements 

.762 

.509 

xs 

Unconditional  Jumps 

. 54U 

.664 

XI 7 

Scaling/Rounding  Opns . 

-.176 

-.024 

. 

X23 

Address  Variable  Freq. 

386 

. 1 6 8 

X28 

Fixed  Point  Var.  Freq. 

. 60  0 

.242 

X37 

Undefined  Variables 

.4  36 

.367 

X42 

Non-Ueated  Do  Loops 

. 386 

. 179 

X51 

Instr.,  4tb  Level  Do  Loops 

-.374 

.4  24 

X53 

Inatr.  , 6t,i  Level  or  Lower 
Do  Loops 

.602 

.34  1 

Table  6-12, 

XI 

Source  Instructions 

.384 

. 59  3 <^rxy 

R*«. 87 1 

X59 

Comment  Statomont/SI 

.2  59 

.203  -.304 

X6  2 

Unconditional  Jumps/Si 

.612 

.5-3  -.121 

X6  3 

Calls/'Linka/ SI 

.196 

-.052 

X71 

Scaling/Rounding  Opns. /SI 

-.261 

-.406  -.382 

A 7 7 

Address  Vir.  Free. /SI 

-.133 

-.022  -.190 

X82 

Fixou  Pc,  Var.  Freq.  SI 

.369 

-.053  -.300 

X96 

Hon-Nasted  Do  Loopy /SI 

-.302 

-.393  -.572 

X9  7 

Do  Loops  Nested  At  2nd  Lvl.  SI 

. ! 62 

.947 

XI  07 

Inatr.,  6tn  Level  or  Lower 
Do  Loops.  SI 

.2  12 

.224  -.117 
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whereas  a much  larger  group  of  programmers  with  varying 
levels  of  skill  and  experience  were  needed  to  develop  the 
499  programs  of  project  M and  B,  since  both  projects  over- 
lapped and  were  concurrent  with  each  other  at  different  stages 
in  their  software  development. 

For  any  of  the  explanations  presented  here,  it  should  be 
kept  in  mind  that  each  of  these  projects  was  functionally 
different  from  the  others,  and  each  was  programmed  in  a 
special  purpose  programming  language.  Any  of  these  factors 
in  addition  to  others  (e.g.  small  sample  size  of  project  P, 
definition  of  variables,  testing  considerations  for  each  pro- 
ject) taken  singly  or  in  combination,  could  be  largely 
responsible  for  these  obtained  results. 

In  summary,  with  the  exception  of  project  P,  the  normalized 
variables  contributed  appreciably  less  to  the  prediction  of 
errors  than  did  the  unnormalized  variables  whose  results  were 
discussed  in  Section  6.3.1.  This  low  predictability  suggests 
the  need  to  identify  any  non-linear  relationships  that  exist 
among  the  predictors  and  errors,  and  then  to  define  accordingly 
the  most  appropriate  non-linear  model  which  could  be  used  to 
improve  these  predictors.  Given  these  results,  no  prediction 
of  errors/program  was  attempted  using  selected  sets  of  normalized 
and  unnormalized  predictors  in  combination. 

6.3.3  Error  Rate/Program  = f (Unnormalized  Variables) 

As  discussed  earlier  in  Section  5.0,  the  newly  constructed 
dependent  variable,  error  rate/program,  is  analyzed  in  this 
study  both  as  a function  of  the  unnormalized  variables  and  as 
a function  of  Source  Instruction  plus  the  unnormalized  variables. 
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The  prediction  results  using  only  the  unnormalized  variables 
are  presented  in  Tables  6-16  thru  6-20.  Tables  6-21  and  6-22 
present  summaries  of  these  results  for  five  and  eight  predic- 
tors, respectively.  The  summary  for  eight  predictors  is  pre- 
sented based  on  the  lowest  maximum  number  of  variables  to  be 
entered  in  the  equation  over  all  projects.  The  lowest  maximum 
value  was  eight  for  project  P for  this  set  of  predictors.  Here, 
as  in  Section  6.3.1,  the  number  of  unnormalized  predictor 
variables  that  were  made  available  for  automatic  selection 
for  entry  in  the  regression  equation  were  45  each  for  projects 
M and  P and  43  for  project  B (These  variables  are  listed  in 
Table  C-l  of  Appendix  C) . 

The  summary  results  clearly  indicate  that  a very  low  level 
of  predictability  (R  values  ranged  from  .10  to  .30)  is  obtained 
for  error  rate  using  the  unnormalized  variables.  This  predic- 
tion is  consistently  low  over  each  of  the  regression  equations 
obtained  for  each  project  both  for  five  and  eight  predictors, 

with  the  exception  of  the  eight  predictor  equation  for  project  P 

2 

which  yields  a moderate  prediction  of  R =.535.  Also,  little 
consistency  among  predictors  is  apparent  across  all  three 
projects.  When  eight  predictors  were  used  only  two  predictors, 
M1',  Using  Instructions  and  MS,  Comment  P tutements , appeared  in 
the  results  for  all  projects.  For  five  predictors.  Using 
Instructions  was  the  only  variable  consistently  selected  in  the 
equation  for  each  project. 

One  interesting  result  observed  for  this  set  of  predictions 
was  that  when  zero  error  rate  programs  in  project  M and  B were 

deleted  from  the  analysis  (see  Tables  6-17  and  6-19),  the  pre- 

2 2 
dictions  improved;  R for  project  f!  increased  .06,  and  R for 

project  B increased  .12.  In  the  two  previous  sections,  the 
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TABLE  6-16.  PROJECT  M, 

ERROR  RATE/PROGRAM  = f( UNNORMAL  I ZED  VARIABLES) 


VARIABLES  IN 

PREDICTION 

EQUATION 

i 

8 

9CMAX. ) 

MULTIPLE  R, 

. 291 

.314 

.318 

MULTIPLE  R‘ 

.085 

. 09  9 

.101 

STD.  ERROR  OF  ESTIMATE 

2.890 

2.879 

2.879 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

XI  SOURCE  INSTRUCTIONS 

.722 

- .820 

- .131" 

X4  USING  INSTRUCTIONS 

.251 

. 243 

.041 

X5  COMMENT  STATEMENTS 

— 

.390 

- .028 

XI 5 CONDITIONAL  JUMPS 

.370 

. 277 

- .090 

X16  FUNCTIONS 

.220 

.219 

.033 

X35  REGISTER  VARIABLES 

.133 

- .105 

- .048 

X37  UNDEFINED  VARIABLES 

— 

.148 

- .047 

X45  DO  LOOPS  NESTED  AT  4TH  LEVEL 

— 

.074 

- .050 

ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

£ 

5 

REGRESSION 

301.531 

5 

60.306 

7.221 

<.001 

RES  I DUAL 

3248.768 

339 

8.352 

8 

REGRESSION 

350 . 229 

8 

43.779 

5.281 

<.001 

RESIDUAL 

3200.070 

386 

8 .290 

TOTAL 

3550 . 299 

394 

" BEST  SINGLE  PREDICTOR 
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TABLE  6-17.  PROJECT  M, 

ERROR  RATE/PROGRAM  = f( UNNORMAL  I ZED  VARIABLES), 
ZERO  ERROR  RATES  DELETED 


VARIABLES  IN 

PREDICTION 

EQUATION 

1 

8 

11CMAX.) 

MULT 

IPLE  R 

. 371 

.394 

.410 

MULT 

IPLE  R^ 

.138 

.155 

.168 

STD. 

ERROR  OF  ESTIMATE 

2.877 

2.862 

2.855 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

XI 

SOURCE  INSTRUCTIONS 

-1.080 

-1.375 

- . 292” 

Xll 

USER  MACROS 

.099 

.096 

- .123 

XI  5 

CONDITIONAL  JUMPS 

.421 

.356 

- .211 

X16 

FUNCTIONS 

.202 

.174 

- .106 

X27 

FIXED  PT.  VARIABLES 

— 

- .099 

- .201 

X38 

UNDEFINED  VAR.  FREQ. 

.250 

.210 

- .2  20 

X45 

DO  LOOPS  NESTED  AT  4TH  LEVEL 



- .083 

- .100 

X54 

SI  X AVG.  NO.  OPERATORS/ 

— 

.469 

- .262 

ARITHMETIC  INSTR. 


ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

5 REGRESSION 

3 9 6.0  1 5 

5 

79.203 

9.571 

<.001 

RESIDUAL 

2474.430 

299 

3.276 

8 REGRESSION 

446.074 

8 

55  .759 

6 .808 

<.001 

RESIDUAL 

2424.371 

296 

8.190 

TOTAL 

2870.445 

304 

::  BEST  SINGLE  PREDICTOR 
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TABLE  6-18.  PROJECT  B, 

ERROR  RATE/PROGRAM  = f (UNNORMALI ZED  VARIABLES) 


VARIABLES  IN  PREDICTION  EQUATION 


1 

8 

9 (MAX. ) 

MULTIPLE  R, 

.362 

.418 

.430 

MULTIPLE  R i 

.131 

.175 

.185 

STD.  ERROR  OF  ESTIMATE 

2.380 

2.356 

2.354 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

X5 

COMMENT  STATEMENTS 

— 

.162 

- .105 

XI 5 

CONDITIONAL  JUMPS 

- .481 

- .568 

- . 2 3 4 5{ 

X 1 7 

SCALING/ROUNDING  OPNS . 

. 143 

.212 

- .024 

X37 

UNDEFINED  VARIABLES 

.358 

.371 

- .121 

X49 

INSTR.,  2ND  LEVEL  DO  LOOPS 

.223 

.307 

- .037 

X50 

INSTR.,  3RD  LEVEL  DO  LOOPS 

— 

- .236 

- .179 

X52 

INSTR,,  5TH  LEVEL  DO  LOOPS 

— 

.173 

.056 

X54 

SI  X AVG.  NO.  OPERATORS/ 

- .339 

- .424 

- .217 

ARITHMETIC  INSTR. 


ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

P 

5 

REGRESSION 

83.803 

5 

16.761 

2.959 

<.05 

RESIDUAL 

555.173 

98 

5.665 

8 

REGRESSION 

111.506 

8 

13.938 

2.510 

<.05 

RESIDUAL 

527 . 470 

95 

5.552 

TOTAL 

633.976 

103 

BEST  SINGLE  PREDICTOR 
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TABLE  6-19.  PROJECT  B, 

ERROR  RATE/PROGRAM  = f (UNNORMAL I ZED  VARIABLES), 
ZERO  ERROR  RATES  DELETED 


VARIABLES  IN  PREDICTION  EQUATION 


MULTIPLE  R 
MULTIPLE  R 

STD.  ERROR  OF  ESTIMATE 


1 8 1 5 CMAX . ) 

.484  .543  .629 

.234  .295  .393 

2.144  2.093  2.022 


VARIABLES  CX) 

COEFFICIENTS 

rxy 

X4 

USING  INSTRUCTIONS 

.371 

.445 

.084 

X 1 0 

SYSTEM  MACROS 

— 

- .292 

- 

.212 

Xll 

USER  MACROS 

.328 

.438 

- 

.166 

X 2 8 

FIXED  PT.  VAR.  FREQ. 

.277 

— 

- 

.052 

X43 

DO  LOOPS,  NESTED  AT  2ND  LEVEL 

— 

.383 

- 

.121 

X47 

DO  LOOPS,  6 TH  LEVEL  OR  LOWER 

— 

.428 

- 

.070 

X49 

INSTR.,  2ND  LEVEL  DO  LOOPS 

.316 

.755 

- 

.082 

X 50 

INSTR.,  3RD  LEVEL  DO  LOOPS 

--- 

- .342 

- 

.211 

X 5 4 

SI  X AVG.  NO.  OPERATORS/  -1.162 

ARITHMETIC  INSTR. 

-1  .022 

- 

.310” 

ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

P 

5 

REGRESSION 

125.202 

5 

25.040 

5.447 

<.001 

RES  I DUAL 

409 .144 

39 

4.597 

8 

REGRESSION 

157.667 

3 

19.708 

4.500 

<.001 

RESIDUAL 

376.630 

36 

4.38(! 

TOTAL 

534.346 

94 

i:  BEST  SINGLE  PREDICTOR 
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TABLE  6-20.  PROJECT  P, 

ERROR  RATE/PROGRAM  = f (UNNORMAL I ZED  VARIABLES) 


VARIABLES  IN  PREDICTION  EQUATION 


MULTIPLE  R,  .643 
MULTIPLE  R^  .413 
STD.  ERROR  OF  ESTIMATE  1.048 


NO. 

PREDICTORS 


SUM  OF 
SQUARES 

OF 

MEAN 

SQUARE 

F 

E- 

REGRESSION 

22.427 

5 

4.485 

4.087 

<.01 

RESIDUAL 

31.824 

29 

1.097 

REGRESSION 

29.002 

8 

3.625 

3 . 733 

<•01 

RESIDUAL 

25.249 

26 

.971 

TOTAL 

54.251 

34 

BEST  SINGLE  PR  DICTOR 
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TABLE  6-21. 


FIVE  PREDICTOR  SUMMARY, 

ERROR  RATE /PROGRAM  s f ( UNNORMAL  I ZED  VARIABLES) 


VARIABLES 

Ma 

Mb 

8* 

3U 

P 

REGRESSION  COEFFICIENTS 

X4  USING  INSTRUCTIONS 

. 251 

.371 

- .319 

X15  CONDITIONAL  JUMPS 

.3  7.1 

.421  ' 

- .481" 

XI  SOURCE  INSTRUCTIONS 

- . 722!' 

-1.080" 

XII  USER  MACROS 

.199 

.328 

XI 6 functions 

.221 

. 212 

X35  REGISTER  VARIABLES 

- .133 

- .276 

X49  INSTR.  2ND  LEVEL 

.223 

.316 

DO  LOOPS 

X54  SI  X AVG.  NO,  OPERATORS/ 

- .339 

-1.162" 

ARITHMETIC  INSTR. 

>2  ENTRY  POINTS 

- .499" 

CORRELATION  STATISTICS 

r",SI 

1 .111 

1.J11 

.902 

.979 

.263 

rSI , ERROR  RATE 

- .131 

- .292 

- .195 

- .276 

- .301 

r!«,  ERROR  RATE 

- .131 

- .29? 

- .234 

- .310 

- .322 

r», ERROR  RATE 

.117 

.085 

.055 

,0y6 

. 104 

PREDICTION  SUMMARY 

K 

.291 

.171 

. 362 

. 484 

.643 

R2 

.18  5 

.138 

. 131 

. 234 

.413 

"all  observations  used 

bZERO  ERROR  RATES  DELETED 


-BEST  SINGLE  PREDICTOR 


TABLE  6-22 


BIGHT  PREDICTOR  SUMMARY, 

ERROR  RATE/PROGRAM  a f CUNNORMAL I ZED  VARIABLES) 


VARIABLES 

Ma 

Mb 

P 

REGRESSION  COEFFICIENTS 

XL  USING  INSTRUCTIONS 

. 243 

.445 

-1.172 

X5  COMMENT  STATEMENTS 

.090 

.162 

1.105 

XI  5 CONDITIONAL  jumps 

.277 

.356 

- .568“ 

X 5 4 SI  X AVG.  NO.  OPERATORS/. 

.469 

- .424 

- 1 . 02  2“ 

ARITHMETIC  INSTR. 

XI  SOURCE  INSTRUCTIONS 

- .820" 

-1.375“ 

XII  USER  MACROS 

.096 

.438 

X 1 6 FUNCTIONS 

. 219 

.174 

XJ7  UNDEFINED  VARIABLES 

. 143 

.371 

X45  DO  LOOPS,  4TH  LEVEL 

.074 

- .083 

X49  INSTR.,  2ND  LEVEL  DO 

LOOPS 

.30  7 

.755 

X 5 J INSTR.,  3RD  LEVEL  DO 

LOOPS 

- .236 

- ,342 

X 3 EXIT  POINTS 

- ,438" 

CORRELATION  STATISTICS 

r:i  S i 

1.000 

1.00)1 

.90  2 

.979 

. 279 

rSI  .ERROR  RATE 

- .131 

- .292 

- .195 

- .2  76 

- .331 

r-<,  ERROR  RATE 

- .131 

- .29  2 

- .234 

- .310 

- .309 

f ",  ERROR  RATE 

.017 

.03  5 

.05  5 

.096 

.095 

PREDICTION  SUMMARY 

R 

.314 

. 39  4 

.418 

.543 

.731 

0 
R “ 

. 099 

.15  5 

.175 

. V)"j 

.535 

aAL L OBSERVATIONS  USED 
bZER0  ERROR  RATES  DELETED 
:lEEST  SINGLE  PREDICTOR 
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deletion  of  these  same  programs  always  increased  the  value 
2 

of  R for  project  B and  decreased  or  had  little  effect  on  the 
2 

R values  for  project  M,  when  errors/program  was  being  pre- 
dicted. In  these  results,  however,  when  predicting  error  rate 

2 

the  R value  for  project  M increases  approximately  5771  (from 

2 

.099  to  .155),  and  the  R value  for  project  B increases  approxi- 
mately 683  (from  .175  to  .295).  Both  of  these  changes  are 
statistically  significant.  In  fact,  comparatively  speaking, 
when  predicting  error  rate  with  the  zero  observations  removed 
from  the  analysis,  one  observes  that  the  resultant  increase 
in  R values  could  possibly  be  equated  with  the  increase  that 
would  result  from  three  to  four  additional  variables  being 
added  to  the  prediction  equation. 

The  increases  observed  here  in  the  predictability  of 
error  rate  support  the  hypothesis  stated  in  section  3.0  which 
suggested  that  programs  with  zero  reported  errors  actually 
had  latent  errors  still  in  the  program  that  had  gone  undetected 
due  to  a lack  of  thoroughness  in  testing.  As  such,  these 
zero  reported  errors  were  considered  as  underestimates  of 
the  total  errors  in  the  program,  and  their  presence  in  the 
analysis  would  therefore  reduce  the  predictability  of  the  depen- 
dent variable  and  increase  the  predictability  when  they  were 
removed.  More  will  be  said  about  the  error  rate  measure  and 
this  hypothesis  later  in  the  report. 

As  noticed  in  the  previous  section,  when  zero  error  rate 
programs  are  deleted,  different  prediction  equations  result. 

For  project  M,  deleting  the  90  zero  observations  results  i. 
only  three  of  the  eight  predictors  being  reselected;  for 
project  B,  only  three  of  the  eight  predictors  were  reselacted. 
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For  the  be3t  single  predictors  of  error  rate  among  the 

unnormalized  variables  that  were  in  the  regression  equation, 

no  consistency  at  all  is  found  across  each  project  (see  Tables 

6-21  and  6-22).  It  is  interesting  to  note  at  this  time  that 

several  of  the  equations  that  were  developed  for  this  set  of 

predictions  do  not  contain  the  best  single  predictor  of  error 

rate  among  the  total  set  of  unnormalized  variables.  For 

example,  for  project  M using  all  395  observations  (see  Table 

6-16),  the  best  single  predictor  is  X7,  Arithmetic  Instructions 

(rx?  error  rate  ■ -.148)  and  not  XI,  Source  Instructions 

(r  1 error  rate  “ -.131).  XI  is,  however,  the  best  predictor, 

as  reported  in  Table  6-18,  when  the  90  zero  error  rates  are 

deleted  from  the  analysis  for  project  II.  For  the  results  of 

project  B presented  in  Table  6-19  (95  observations  used) , the 

best  predictor  is  X15  Conditional  Jumps  (r,<15  error  rate  " 

-.331)  . Finally,  for  the  results  of  project  P using  five  and 

eight  predictors  (see  Table  6-20) , variable  X2,  Entry  Points 

(r  o ■ -.  322)  is  the  best  predictor  of  error  rate, 

x*  f error  r « to 

The  fact  that  some  of  those  best  single  predictor  vari- 
ables were  not  present  in  die  various  prediction  equations 
that  were  generated  accentuates  the  need  to  proceed  with  caution 
when  any  attempt  is  made  to  decipher  the  relative  contributions 
of  each  variable  to  the  prediction  from  a multiple  regression 
analysis.  Clearly  it  is  the  combination  of  variabJes  which 
should  be  considered  and  not  any  one  variable  separate  from 
the  others. 

In  summary,  for  this  set  of  predictions,  with  the  exception 
of  the  moderate  prediction  of  error  rate  obtained  for  project  P, 
it  is  found  that  linear  combinations  of  unnormalized  variables 
produce  relatively  low  level  predictions  for  error  rate.  This 
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appears  to  be  due  to  the  low  correlations  that  each  of  the 
selected  predictors  had  with  error  rate  (see  Tables  6-16  to 
6-19).  These  low  correlations  stand  in  stark  contrast  to  the 
moderate  to  high  correlations  that  these  same  unnormalized 
variables  had  with  errors.  This  shows  that  when  the  effect 
of  Source  Instructions  is  removed  from  the  error  variable, 
the  normalized  error  variable  (error  rate)  has  little  left  in 
common  with  any  of  the  unnormalized  predictors. 

Thus  it: seems  clear  that  these  results  reiterate  and 
demonstrate  the  fact  that  a large  portion  of  the  relationship 
between  the  unnormalized  variables  and  errors/program  result 
from  the  combined  influence  of  Source  Instructions  in  each. 

This  is  what  was  found  for  the  results  as  presented  in  Section 
6.3.1,  errors  as  a function  of  the  unnormalized  variables. 

It  appears  warranted,  then,  that  any  future  predictions  of  error 
rate  from  combinations  of  the  unnormalized  variables  should 
investigate  non-linear  models,  or  use  variables  other  than  those 
investigated  in  this  study. 


6.3.4  Error  Rate/Program  ■ £(SI  + Normalized  Variables) 


The  final  set  of  prediction  equation  results  are  presented 
in  Tables  6-23  thru  6-27  for  five  and  ten  predictors,  wherein 
error  rate  is  being  predicted  from  a linear  combination  of  the 
normalized  variables  and  Source  Instructions.  Summary  Tables 
6-28  and  6-29,  for  five  and  ten  predictors,  are  also  provided. 

The  number  of  predictors  that  were  made  available  for  selection 
by  the  regression  procedure  were  45  each  for  projects  M and  P 
and  43  for  project  B (see  Table  C-l) . 

In  viewing  the  summary  results  one  can  observe  that  the 

use  of  ten  versus  five  predictors  substantially  improves  the 

predictions  for  each  project.  The  percentage  increase  in 
2 

values  of  R for  each  project  ranged  from  an  increase  of  IS “S 

for  project  P to  an  increase  of  2375  and  381,  respectively,  for 

projects  M and  B.  Additionally,  the  predictability  of  error 

2 

rate  using  this  set  of  predictors  (R  values  ranged  from  .34 

to  .47  for  10  predictors)  is  low,  but  generally  higher  than  the 

previous  results  obtained  for  error  rate  (i.e.,  in  Section  6.3.3) , 

Again  project  P is  an  exception.  For  project  P the  percent 

2 

of  variance  accounted  for  by  five  predictors  is  R ■ .81; 

2 

for  ten  predictors  R ■ .94.  These  results  represent  the  highest 
2 

values  of  R obtained  for  project  P over  all  the  prediction 
equations  that  were  developed  for  this  project. 

Using  ten  predictors,  variable  X58,  Using  Instructions/SI , 
is  the  only  predictor  that  appears  consistently  in  each  of  the 
prediction  equations  for  all  projects.  For  two  of  the  three 
projects,  M and  B,  this  variable  is  also  the  best  single 
predictor  of  error  rate.  For  these  two  projects  Using  Instruc- 
tions/SI accounts  for  47f,  (for  project  B)  and  54S  (for 
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TABLE  6-23.  PROJECT  M, 


ERROR  RATE/PROGRAM  = 

fCSI  + 

NORMALIZED 

VARIABLES) 

VARIABLES  IN  PREDICTION 

EQUATION 

5 

i£ 

16CMAX.) 

MULTIPLE  R, 

.545 

.603 

.625 

MULTIPLE  R i 

. 297 

.364 

.390 

STD.  ERROR  OF  ESTIMATE 

2.532 

2.426 

2.393 

VARIABLES  <X) 

COEFFICIENTS 

r 

xy 

X56  ENTRY  POINTS /S 1 

— — — 

. 244 

. 229 

X57  EXIT  POINTS/S  I 

— 

- .366 

.123 

X58  USING  INSTRUCTIONS/SI 

.350 

. 281 

. 39  7" 

X 5 9 COMMENT  STATEMENTS /S I 

— 

. 179 

. 341 

X64  SYSTEM  MACROS/SI 

— 

.173 

.279 

X70  FUNCTIONS/SI 

. 257 

. 246 

. 245 

X81  FIXED  POINT  VARIABLES/SI 

— 

- .114 

- .024 

X87  LABELED  ARRAY  VARIABLES/SI 

- .129 

- .106 

- .116 

X90  REGISTER  VARIABLES 

- .165 

- .118 

- .165 

FREQ. /SI 

X91  UNDEFINED  VARIABLES/SI 

. 167 

.116 

.320 

ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

£ 

5 

REGRESS  ION 

1055.927 

5 

211.185 

32.935 

<.001 

RESIDUAL 

2494. 371 

339 

6.412 

10 

REGRESSION 

1290.469 

15 

129.047 

21 .928 

<.001 

RESIDUAL 

2259 . 830 

334 

5.885 

TOTAL 

3550 . 299 

394 

::  BEST  SINGLE  PREDICTOR 
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TABLE  6-24 


• PROJECT  M, 

ERROR  RATE/PROGRAM  = f(SI  + NORMALIZED  VARIABLES), 
ZERO  ERROR  RATES  DELETED 


VARIABLES  IN 

PREDICTION 

EQUATION 

5. 

il 

22CMAX. ) 

MULTIPLE  R, 

.610 

.644 

.679 

MULTIPLE  R^ 

.111 

.415 

.462 

STD.  ERROR  OF  ESTIMATE 

2.454 

2.390 

2.341 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

X53  USING  INSTRUCTIONS/SI 

.319 

.273 

.471“ 

X60  LABELED  I NSTRUCT I ONS /S  I 

.105 

.221 

.227 

X64  SYSTEM  MACROS/SI 

. 202 

.199 

.374 

X66  EQUATE  STATEMENTS /S I 

— 

- .092 

- .051 

X68  LOGICAL  CONNECTORS/SI 

— 

.137 

.107 

X70  FUNCTIONS/SI 

. 192 

.173 

.174 

X73  NESTED  SHORT  DO  LOOPS/SI 

— 

- .096 

- .030 

X31  FIXED  POINT  VARIABLES/SI 

— 

- .082 

- .020 

X84  FLOATING  PT.  VARIABLE 

— 

.104 

- .103 

FREQ. /SI 

X91  UNDEFINED  VARIABLES/SI 

.214 

.277 

.415 

ANALYSIS  OF  VARIANCE 

NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

P 

5 

REGRESSION 

1069.392 

5 

213.878 

35.507 

<.001 

RESIDUAL 

1801.053 

299 

6.024 

10 

REGRESSION 

1190.639 

10 

119.069 

20.840 

<•001 

RESIDUAL 

1679.755 

294 

5.713 

TOTAL 

2870.445 

304 

::  BEST  SINGLE  PREDICTOR 
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TABLE  6-25. 


PROJECT  B, 

ERROR  RATE/PROGRAM  =f(SI  + NORMALIZED  VARIABLES) 


VARIABLES  IN 

PREDICTION 

EQUATION 

5. 

1£ 

13CMAX.) 

MULTIPLE  R- 

. 494 

.579 

.605 

MULTIPLE  R 

. 244 

.336 

.365 

STD.  ERROR  OF  ESTIMATE 

2.221 

2.137 

2.123 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

X57 

EXIT  POINTS/SI 

. 141 

.362 

.219 

X58 

USING  INSTRUCTIONS/SI 

. 30  4 

.363 

.312“ 

X61 

ARITHMETIC  I NS TRUCT 1 ONS / S I 

— 

- .139 

- .069 

Xb  4 

SYSTEM  MACROS/SI 

— 

- .391 

.041 

X6  8 

LOGICAL  CONNECTORS/SI 

--- 

.158 

.038 

X91 

UNDEFINED  VARIABLES/SI 

— 

.158 

.140 

X9  7 

DO  LOOPS  NESTED  AT  2ND 
LEVEL/SI 

— — — 

- .270 

.114 

X99 

DO  LOOPS  NESTED  AT  4TH 
LEVEL/SI 

- .653 

- .680 

- .211 

X103 

INSTR . , 2ND  LEVEL  DO 
LOOPS/SI 

. 241 

.399 

.191 

X105 

INSTR.,  4TH  LEVEL  DO 
LOOPS/SI 

.508 

.525 

- .130 

ANALYSIS  OF 

VARIANCE 

NO. 

SUM  OF 

MEAN 

PREDICTORS 

SQUARES 

DF 

SQUARE 

F 

b 

5 

REGRESSION 

15  5 . 766 

5 

31.153 

6.318 

<•00  1 

RESIDUAL 

483.210 

98 

4.931 

10 

REGRESSION 

214.411 

10 

71,441 

4.697 

<.001 

RESIDUAL 

424.565 

93 

4.565 

TOTAL 

638.978 

103 
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TABLE  6-26.  PROJECT  B, 

ERROR  RATE/PROGRAM  s f(SI  + NORMALIZED  VARIABLES), 
ZERO  ERROR  RATES  DF.LETED 


VARIABLES  IN 

PREDICTION 

EOUAT I ON 

MULTIPLE  R, 

MULTIPLE  R 1 

STD.  ERROR  OF  ESTIMATE 

5 

.614 

.376 

1.935 

11 

.683 

.467 

1.842 

23CMAX. ) 

.775 

.600 

1.734 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

X57 

EXIT  POINTS/SI 

... 

.199 

.284 

X58 

USING  INSTRUCTIONS/SI 

.386 

. 480 

.467" 

X60 

LABELED  I NS TRUCT I ONS /S I 

- -- 

- .237 

.037 

X6  2 

UNCONDITIONAL  JUMPS/SI 

— 

.228 

- .090 

X66 

EQUATE  STATEMENTS/SI 

- .232 

- .160 

- .232 

X70 

FUNCTIONS/SI 

--- 

- .205 

- .090 

X71 

SCALING/ROUNDING  OPNS./S 

I 

.232 

.035 

X91 

UNDEFINED  VARIABLES/SI 

.241 

.360 

.193 

XI 0 3 

INSTR.,  2ND  LEVEL  DO 
LOOPS /S I 

.259 

.223 

.216 

XI 0 8 

SI  X AVG.  NO.  OPERATORS/ 

- .182 

- . 148 

- .143 

ARITHMETIC  INSTR./SI 


ANALYSIS  OF  VARIANCE 


NO. 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

E 

5 

REGRESSION 

201.097 

5 

40.219 

10.741 

<.001 

RESIDUAL 

333.249 

89 

3.744 

10 

REGRESSION 

249.335 

10 

24.933 

7.349 

<•001 

RESIDUAL 

2 8 5.0  1 2 

84 

3.393 

TOTAL 

534.346 

94 

” BEST  SINGLE  PREDICTOR 
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TABLE  6-27 


PROJECT  P, 

ERROR  RATE/PROGRAM 


f (SI 


+ 


NORMALIZED  VARIABLES) 


VARIABLES  IN 

PREDICTION 

EQUATION 

i 

11 

20 (MAX. 0 

MULTIPLE  R, 

.901 

.969 

.994 

MULTIPLE  RZ 

.812 

.939 

.989 

STD.  ERROR  OF  ESTIMATE 

.593 

.373 

.209 

VARIABLES  (X) 

COEFFICIENTS 

rxy 

X58 

USING  l NSTRUCT I ONS /S I 

— 

- .316 

- .176 

X59 

COMMENT  STATEMENTS /SI 

.606 

.467 

.813“ 

X61 

ARITHMETIC  INSTRUCTIONS/SI 

— 

.530 

.009 

X6  2 

UNCONDITIONAL  JUMPS/SI 

. 360 

.336 

.421 

X69 

CONDITIONAL  JUMPS/SI 

— 

.143 

.209 

X74 

LOCK  MACROS/SI 

- .260 

— 

- .020 

X84 

FLOATING  PT.  VAR.  FREQ. /SI 

- .300 

- .181 

X9  7 

DO  LOOPS  NESTED  AT  2ND 
LEVEL/SI 

.309 

. 5 5 5. 

.552 

X9  8 

DO  LOOPS  NESTED  AT  3RD 
LEVEL/SI 

• — - 

- .285 

.169 

XI 0 2 

INSTR.  IN  NON-NESTED 
DO  LOOPS/SI 

- .265 

- .535 

.031 

XI 0 5 

INSTR.  IN  4TH  LEVEL 
DO  LOOPS/SI 

- .255 

- .121 

ANALYSIS  OF  VARIANCE 


NO  . 

PREDICTORS 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARE 

F 

P 

5 

REGRESSION 

44.038 

5 

8.838 

25.008 

<.001 

RESIDUAL 

10.214 

29 

.352 

10 

REGRESSION 

50.918 

10 

5.092 

36.667 

<.001 

RESIDUAL 

3.333 

24 

.139 

TOTAL 

54.251 

34 

- BEST  SINGLE  PREDICTOR 


FIVE  PREDICTOR  SUMMARY 
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aALL  OBSERVATIONS  USED 
bZERO  ERROR  RATES  DELETED 
-BEST  SINGLE  PREDICTOR 


TEN  PREDICTOR  SUMMARY, 

ERROR  RATE /PROGRAM  s f(Sl  + NORMALIZED  VARIABLES) 


VARIABLES 


REGRESSION  COEFFICIENTS 


x 5 a 

USING  INSTRUCT  IONS /SI 

. 2 8 1 !l 

. 2 7 8i‘ 

. 36  3“ 

.483"  - 

.316 

X91 

UNDEFINED  VARIABLES/SI 

, 115 

.277 

.158 

.363 

X57 

EXIT  PO 1 NTS /S 1 

. 555 

. 552 

. 199 

X6  4 

SYSTEM  MACROS/SI 

.173 

. 199  - 

.391 

X 7 ft 

FUNCTIONS/SI 

. 246 

.178 

- 

.23  5 

X 59 

COMMENT  STATEMENTS/SI 

.179 

.467 

X6  3 

LABELED  I NSTRUCT I ONS /S I 

.221 

- 

. 237 

X 6 1 

ARITHMETIC  INSTRUCTIONS/SI 

- 

.139 

.533 

X6  2 

UNCONDITIONAL  JUMPS/SI 

.223 

.336 

X6  6 

EQUATE  STATEMENTS/SI 

- 

.1192 

- 

. 163 

X68 

LOGICAL  CONNECTORS/SI 

. 1 57 

.158 

X 8 1 

FIXED  PT.  VARIABLES/St  - 

.114  - 

, ,182 

X84 

FLOATING  PT.  VAR. 

. 1,14 

- 

.33  3 

FREQ. /SI 

X9  7 

DO  LOOPS,  2ND  LEVEL/SI 

- 

.273 

.555 

X 1 j 3 

INSTR.,  2ND  LEVEL  DO 

.399 

.223 

LOOPS/SI 
CORRELATION  STATISTICS 

rS I, ERROR  RATS 
*>>,  ERROR  RATE 
1 ",  ERROR  RATE 

PREDICTION  SUMMARY 


4all  observations  used 

bZERO  ERROR  RATES  DELETED 
“'BEST  SINGLE  PREDICTOR 


.131 

- .292 

- .195 

- .276 

- .301 

.39  7 

.471 

.312 

.457 

.813 

.158 

.222 

.397 

.213 

'.663 

.63  3 

.644 

.579 

.63  3 

.969 

. 36  4 

.415 

.336 

.46  7 

.939 
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project  M)  of  the  total  variance  accounted  for  by  all  10 
predictors  selected  for  each  project.  In  addition,  for  all 
projects,  the  normalized  program  complexity  variables  are  also 
contributing  significantly  to  the  predictions  for  error  rate 
in  these  results. 

Although  the  correlation  of  Using  Instructions/Sl  with 
error  rate  is  low  (from  .31  to  .47)  for  those  projects  where 
it  is  the  best  predictor,  the  fact  that  this  variable  does 
contribute  substantially  to  error  rate  predictions  is  important 
to  consider.  Using  Instructions  as  described  in  Table  B-1  are 
instructions  used  to  establish  data  structure  interfaces  in 
the  program.  An  interpretation  then  for  the  normalized  vari- 
able, Using  Instructions/SI , is  the  number  of  instructions  per 
100  lines  of  code  used  to  establish  data  structure  interfaces; 
or  more  simply  interpreted,  data  interfaces  per  100  lines  of 
code.  The  fact  that  interfaces  have  been  found  to  contribute 
to  errors  are  results  which  support  the  findings  of  other 
researchers  (Thayer  et  al.  1976;  and  Okimoto,  1975)  (13,  14), 
as  cited  earlier  in  Section  6.2. 

Once  again  as  found  in  the  immediately  preceding  results 
for  error  rate  as  a linear  function  of  the  unnormalized  vari- 
ables, one  can  observe  that  the  predictions  for  error  rate  are 
consistently  improved  for  projects  M and  B when  the  zero 
program-observations  are  deleted  from  the  analysis.  This  result 
suggests  that  analysis  of  error  rate  supports  the  hypothesis  of 
lack  of  thoroughness  of  testing  in  some  of  these  programs. 

An  interesting  finding  in  these  results  is  that  variable 
X59,  Comment  Statements/SI , is  the  best  single  predictor  for 
error  rate  for  project  P.  The  fact  that  X59  alone  accounts 
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for  approximately  70S  of  the  total  variance  accounted 
for  by  all  ten  predictors  selected  for  this  project  is  sur- 
prising. This  percentage  was  even  higher,  81%,  when  five 

variables  are  considered. 

The  fact  that  the  results  for  project  P have  been  so 
consistently  different  from  the  results  of  projects  M and  B 
suggests  that  the  function  being  programmed  for  project  P,  the 
software  development  and  testing  environment,  the  programmers 
and  management  of  project  P,  and  the  CENTRAN  language  itself 
are  all  contributing  in  some  distinct  or  interactive  way  to 
produce  these  unique  results.  Possibly  the  fact  that  project 
P implemented  the  new  programming  techniques  whereas  the  other 
projects  did  not  explains  the  high  predictability  of  error  rate 
in  these  results.  Although  it  could  be  suggested  that  structured 
programs  would  require  only  few  comment  statements  due  to  the 
inherent  readability  that  structuring  a program  is  supposed  to 
provide,  the  opposite  could  have  be^n  true  for  these  programs, 
That  is,  since  over  half  (6G3)  of  the  programs  were  structured 
and  more  readable  as  a result,  more  comments  may  have  been 
incorporated  as  a direct  result  of  being  able  to  more  readily 
understand  the  flow  of  logic  and  read  the  programs.  This 
may  have  resulted  in  errors  per  100  lines  of  code  being  more 
easily  detected  when  problems  arose  in  the  programs  . Neverthe- 
less, regardless  of  the  numerous  plausible  hypotheses  that 
could  explain  the  results  for  project  P,  given  the  information 
available,  it  can  only  bo  suggested  that  a variety  of  factors, 
as  stated  above,  nay  account  for  these  results- 


In  summary,  since  the  results  obtained  for  both  sets  of 
predictions  for  error  rate  were  relatively  low  (see  Sections 
6.3.3  and  6.3.4),  no  further  attempt  to  combine  the  normalized 
and  unnormali2ed  variables  to  predict  error  rate  was  made. 

Data  Interfaces  per  100  lines  of  code  is  a significant  contri- 
butor to  the  predictions  for  error  rate.  Other  program 
complexity  variables  also  significantly  contribute  to  these 
predictions.  Error  rate  was  found  to  be  a meaningful  measure 
to  use  for  detecting  the  effects  of  error  free  programs  on  the 
prediction  of  programming  errors.  Finally,  the  low  predictions 
obtained  suggest  the  need  to  analyze  these  variables  with 
non-linear  models . 

6*3*5  Validation  of  Prediction  Equations  for  Sample  S 

The  major  purpose  of  validating  predictions  is  to  identify 
and  examine  how  well  the  same  level  of  predictions  can  be 
maintained,  carried  over,  or  reproduced  for  a separate  data 
sample.  This  separate  sample  is  assumed  to  be  drawn  from  the 
same  population  a3  the  first  sample  for  which  the  prediction 
results  were  obtained.  In  practice,  this  validation  procedure 
(normally  referred  to  as  cross-validation)  is  usually  carried 
out  by  applying  the  obtained  regression  coefficients  to  an 
identical  set  of  predictor  variables  collected  for  the  separate 
data  sample.  Using  these  coefficients  and  the  predictor  vari- 
able values,  estimates  of  the  dependent  variable  can  then  be 
directly  computed.  These  estimates,  when  correlated  with  the 
actual  values  of  the  dependent  variable  that  have  been  collected 
for  the  separate  sample,  can  then  provide  information  about 
how  well  this  separate  sample  validates;  i.e.,  shows  consistency 
of  prediction,  with  the  original  sample.  Validation,  then,  can 
be  considered  as  an  important  means  for  assessing  to  what  extent 
prediction  results  can  be  generalized  to  other  samples. 
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Generally,  It  has  been  found  that  when  the  regression 
coefficients  obtained  from  a multiple  regression  analysis  on 
one  sample  are  applied  to  a second  sample,  the  correlation 
between  the  weighted  predictors  and  the  dependent  variable  in 
the  second  sample  will  be  less  than  the  multiple  correlation 
value  (R)  originally  obtained  from  the  first  sample.  This 
phenomenon  is  referred  to  as  shrinkage  of  the  multiple  corre- 
lation coefficient  (Kerlinger  and  Pedhazur,  pp.  282-284; 

Ferguson,  pp.  401-402)  (4,  8).  Basically,  the  reason  for  this 
is  that  the  multiple  regression  performed  on  the  sample  data 
capitalizes  on  chance.  The  highest  product  moment  correlations 
are  selected  to  enter  into  the  regression  equation,  and  on  subse- 
quent samples  these  correlations  would  probably  be  lower,  there- 
fore yielding  a somewhat  lower  overall  multiple  correlation. 


The  extent  of  the  bias  in  sample  values  of  R is  directly 
dependent  upon  the  population  value  of  the  multiple  correlation 
coefficient,  the  sample  size,  and  the  number  of  predictor 
variables  used  in  the  equations.  For  validation  purposes,  it 
is  possible  to  estimate  the  amount  of  shrinkage  that  will 
result  when  a second  samnle  is  to  be  validated.  The  computa- 
tional formula  used  to  provide  this  estimate  is  as  follows!' 
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(l-p.2) 


N-l  1 

U-k-L 


(6.1) 


x 2 

where  R « the  estimated  squared  multiple  correlation  in  the 
population;  R2  ■ the  obtained  squared  multiple  correlation  for 
the  first  sample  used  to  develop  the  prediction  equation; 

N * the  number  of  observations  in  the  second  (i.e.,  validation) 

sample;  and  k ■ the  number  of  predictor  variables  used  to  obtain 
2 

the  R value. 
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For  sample  S (as  well  as  for  sample  T) , no  separate 

sample  of  programs  were  initially  randomly  selected  and  set 

aside  to  use  for  cross-validation  purposes.  The  interest  at 

that  time  was  to  achieve  the  maximum  attainable  predictions 

and  the  most  representative  results  possible,  using  all  the 

observations  that  had  been  provided  for  each  data  sample. 

Resultantly,  for  validation  of  sample  S predictions,  formula 

(6.0)  was  used  to  estimate  the  amount  of  shrinkage  of  the 

2 

obtained  values  of  R that  would  result  had  samples  for  each 
project  been  set  aside  or  made  available  for  validation. 

2 

Since  the  highest  values  of  R were  obtained  when  errors/ 

program  wore  predicted  as  a function  of  the  unnormalised  variables 

(see  results  of  Section  6.3.1),  only  these  equations  were 

selected  for  validation.  The  validation  results  are  presented 

in  Table  6-30  for  both  five  and  ten  predictors.  The  obtained 
2 

value  of  R , number  of  observations  used,  and  table  reference 
for  the  prediction  equation  results  as  cited  in  Section  6.3.1 
are  presented  at  the  top  of  Table  6-30.  ft2  values  and  shrinkage 
results  are  presented  for  sample  3ises  of  N * 20,  50,  and  150 
programs,  including  a sample  size  equal  to  the  actual  number 
of  observations  used  for  the  original  predictions. 

Clearly  these  results  show  that  the  prediction  equations 

could  be  expected  to  validate?  i.e.,  show  increasing  consistency 

2 

with  the  obtained  values  of  R , as  the  sample  sxze  increases 

and  as  the  ratio  of  predictors  to  observations  in  the  sample 

becomes  smaller.  For  example,  for  the  five  predictor  equation 

for  project  M,  with  a predictor  to  sample  size  ratio  of 
a 2 9 

k/N  **  1:4,  R is  about  two-thirds  the  size  of  R (i.e.,  .239 

and  .514  respectively) . When  the  hypothetical  validation 
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ZERO  ERRORS  DELETED 

NOBS.  = NUMBER  OF  OBSERVATIONS 


TABLE  6-30.  SAMPLE  S,  VALIDATION  RESULTS  FOR  FIVE  AND  TEN  PREDICTOR 
REGRESSION  EQUATIONS,  ERRORS /PROGRAM  = f( UNNORMAL I ZED 
VARIABLES  ([CONTINUED)  - — 


a. 


A 

U0  A CP 
| A 00 
A • 


co  r*. 

rs.  h 

rv 
• • 
I 


a cm 

00  rH 
00  TO 
• * 
I 


cm  a rH  4 

os  to  4 A 4 

oo  to  • • oo  to 

» • rH  • ■ 

I I 


1 


A 


as 


a us  0"» 

I otn  p* 

LA  • 


CM  A 
rs.  cm 
IA  CM 
• • 
I 


A CM 

r>  cm 

rs  m 
» • 
l 


rH  UD  TO  A 4 

oi  to  rH  rs  cm 

rs  to  • • Ps  TO, 

• • H « • 

I I 


<0 

CO 


A* 

4 4 CM 

I TO.  A* 

A rH  • 


4 a rs.  TO, 

CM  TO.  CD  A 

4 m lO  to 

• • » • 

I I 


ON  00  TO.  CO  CD 

rH  TO.  rH  4 CM 

|s,  TO  » • uS  TO. 

• * H ♦ • 


■S: 


A A Ps 
I TO.  IA 
A A • 


A 

cp  rs. 

TO  4 
• • 

I 


4 rs.  oo  m 

CM  4 A rH 

A TO  A TO 

• • • 1 

I I 


rr  lO  A 

A A rH 
• • A TO. 

rH  • • 


ij 


\ 


CP 

CM  A 00 
I 4 A 
A A * 


A JO 
A A 
rH  4 
• • 


4 A 
4 4 
A TO 
• • 
I 


fs.  CM  TO  CO  rH  { 

N H 4 Ds  TO 

A TO  • • A TO  t 

• • rH  ♦ • 1 


I 


t 

0 


LU  ACM 

CM 

UJ 

CM 

U CO  * 

XCL 

o 

z o 

< 

uj  z 

u 

CL 

4L. 

TO 

LU 

*■  ■* 

rH 

lU 

VM 

• • 

UJ 

• • 

^*1* 

CL 

rH 

A 

II 

It 

LU 

4- 

s. 

-1 

•s. 

:u 

CO 

* 

< 

s 

H 

s 

TO 

TO 

rH 

rH 

II 

II 

s 

S 

TO 

TO 

TO 

TO 

rH 

CM 

rH 

II 

II 

M 

z 

2 

oc 

CL 

O' 

o 

o 

O 

u* 

u_ 

U. 

UJ 

CM 

LU 

Z«N 

UJ 

C 

<CL 

O 

c 

< 

< 

< 

u 

V 

TO 

z 

z 

t—4 

A 

r* 

M 

CL 

• • 

CL 

r 

r 

A 

11 

A 

A 

z 

M 

* 

TO 

s 

hH 

TO 

II 

rH 

II 

* 

s 

S 

(/) 

TO 

CO 

TO 

o 

A 

z 

II 

II 

Z 

z 

a 

a 

O 

o 

u. 

U. 

£ 

c 

r* 

< 

> 

C. 

u 


a 

A 

LU 

c: 

1/1 

3 

C 

r> 

UJ 

>- 

L. 

1/1 

•J 

C 

Z 

o 

LU 

c: 

>H 

.0 

u 

1" 

ti 

< 

/> 

« . 

> 

JC 

QC 

3 

LU 

X 

1/1 

X 

n 

33 

UJ 

O 

• 

o 

A 

_! 

u 

Cl 

-1 

LU 

o 

< 

N 

z 

u 

6-59 


i — —..  .I.nitm.ii..  I 


sample  size  is  increased  up  to  N ■ 150  and  k/N  ■ 1:30,  the 

2 

estimated  shrinkage  of  R is  very  small  (approximately  .02) , 

^ 2 

with  an  expected  change  in  R from  .514  to  .499. 

For  additional  discussion  concerning  the  role  of  valida- 
tion procedures  in  multiple  regression  analysis  the  reader 
should  consult  the  following  texts  and  articles  (Kerlingor  and 
Pedhazur,  19 73;  Lord  and  Novick,  1968;  Herzberg,  1969?  and 
Hosier,  1951)  (7,  8,  9,  12). 


6-60 


•r 


..  jt 


'7*m- 


6 . 4 Sample  T Results 


Prediction  equation  results  for  errors  .\nd  error  rate 
using  sample  T programs  are  presented  in  the  following  pages. 
Results  for  five  predictors  were  chosen  primarily  because 
they  were  generally  found  to  provide  almost  as  good  a predic- 
tion of  the  dependent  variable  as  that  obtained  when  the 
statistically  selected  maximum  number  of  variables  had  been 
entered  in  the  equation.  Usually  a maximum  of  six  to  nine 
predictors  were  entered  in  the  equation.  In  several  subsystem 
predictions  for  error  rate,  the  maximum  number  of  predictors 

was  less  than  five.  The  maximum  number  of  predictors  entered 

•%*<•  ........ 

and  tSV multiple  correlation  coefficient,  R,  obtained  using 
these  predictors  are  nevertheless  reported  for  each  subsystem  and 
for  each  dependent  variable.  Due  to  the  large  number  of 
regression  equations  being  presented  in  this  section,  the 
analysis  cf  variance  tables  are  presented  only  for  those  re- 
gressions for  which  the  same  set  of  predictors  in  all  subsystems 
were  available  to  be  used.  Additionally,  as  with  sample  S, 
results  are  reported  for  each  subsystem  using  all  observations 
and  with  zero  errors  deleted  from  the  analysis.  Finally,  all 
results  presented  in  this  section  should  be  interpreted  cau- 
tiously. This  particularly  applies  to  the  results  obtained 
for  (1)  subsystems  B,  D,  and  E , specifically,  because  of  the  very 
limited  number  cf  program-observations  in  each  (see  Table  2-6) , 
and  (2)  subsystems  F and  G,  because  of  the  large  percentage  of 
programs  and  source  code  in  each  subsystem  having  zero  reported 
errors  (again  see  Table  2-6,  Sample  . Subsystem  Statistics). 
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6. 


Results  for  Errors/Program 


The  following  tables  of  prediction  equations  and  other 
results  for  errors/program  are  presented  in  this  section: 


All  observations  used 
^Zero  error  programs  deleted 

CA  maximum  of  23  predictors  were  available  for  selection  in 
these  predictions 

dA  maximum  of  20  predictors  were  available  for  selection  in 
these  predictions. 
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TABLE  6-31.  ERRORS /PROGRAM  = ffPROGRAM  STRUCTURE  + PROGRAMMER  VARIABLES) 


PROGRAMMER  VARIABLES)  (CONTINUED) 
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ERROKS/PCOGRAM  (COtlTIKULD) 
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TABLt  6-36.  BLST  SIUGLE  PREDICTOR  SUMMARY,  EKRORS/PR06RAM  (COUTIKUED) 
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A list  of  sample  T predictor  variables  that  were  candidates 
for  entry  in  the  prediction  equations  for  errors/program 
(i.e.,  Tables  6-31  thru  6-34)  is  presented  in  Table  C-2  of 
Appendix  C. 

In  the  summary  results  (Table  6-35)  for  errors  one  can 

observe  that  over  all  subsystems  of  sample  T,  regardless  of 

whether  or  not  programmer  variables  have  been  entered  in  the 

prediction  equation,  errors/program  is  consistently  highly 

2 

predictable  with  obtained  values  of  R in  the  range  from  .76 
for  subsystem  A to  .99  for  subsystem  D.  Furthermore,  there 
initially  appears  to  be  little  consistency  among  the  best 
single  predictors  of  errors  in  each  subsystem.  However,  con- 
sidering the  high  correlations  that  were  reported  among  the 
five  predictors  TS,  BR,  LS,  DATA,  and  EX,  there  is  a high 
degree  of  commonality  among  the  best  predictors  for  five  of 
the  eight  subsystems  (A,  C,  E,  G,  and  H) . This  commonality  is 
reflected  in  the  correlation  that  each  best  predictor  has  with 
Total  source  Instructions  (TS) , reported  in  Table  6-35  as 
the  value  of  r ( TS  for  each  subsystem.  Note  that  the  best 
predictors  reported  are  those  found  among  only  the  five  pre- 
dictors that  were  in  the  regression  equation.  Sometimes  the 
best  predictor  among  all  variables  to  be  selected  for  entry  in 
the  equation  may  not  appear  in  the  equation  (see,  for  example, 
sample  S results,  section  6.3.3).  Table  6-36  presents  similar 
results  as  in  Table  6-35,  with  the  difference  beinc  that  the 
best  single  predictor  among  all  the  variables  available  for 
selection  has  been  identified. 
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For  these  results  (Table  6-36),  specifically  for  errors 
predicted  from  a linear  combination  of  the  program  structure 
variables  when  the  zero  errors  are  deleted  from  the  analysis, 
the  bust  single  predictors  from  seven  of  the  eight  subsystems 
are  all  found  to  hs.ve  nearly  identical  high  correlations  with 
Total  Source  Instructions.  Although  some  of  the  best  pre- 
dictors are  different  variables,  each  is  basically  reflecting 
the  effect  of  length  of  program  to  a high  degree.  Notice 
also,  in  Tables  6-35  and  6-36,  that  the  best  predictors'  cor- 
relations with  errors  are  generally  moderate  to  high.  As 

indicated  by  the  percent  of  variance  in  errors  accounted  for 

2 

by  the  best  single  predictors  alone  (see  values  of  r 4 

* , errors 

in  Table  6-35) , they  account  for  approximately  515  (sub- 
system c)  to  945  (subsystem  G)  of  the  total  variance  in  errors 

2 

explained  using  all  five  predictors  (i.e.,  the  R values). 
Consequently,  this  suggests  that  other  program  complexity 
variables  are  also  significantly  contributing  to  error  pre- 
dictability in  some  subsystems. 

Referring  to  Table  6-31,  when  errors  are  predicted  using 
both  the  program  structure  and  programmer  variables,  it  is 
observed  that  only  one  programmer  variable,  WKLD,  apnears  in 
any  of  the  equations;  and  then  for  only  two  of  six  subsystems. 
Thus,  the  RAT  and  RAT /WKLD  variables  were  not  selected  as 
statistically  meaningful  variables  for  the  five  predictor 
solutions  being  presented.  When  the  three  programmer  vari- 
ables were  deleted  from  the  analysis  and  errors  were  predicted 
only  as  a function  of  the  program  structure  variables  (see 
Table  6-33),  it  is  observed  that  (1)  the  high  predictability 
of  errors  is  basically  unaffected  and  (2)  the  predictors 
selected  for  each  subsystem  do  not  significantly  differ  from 
those  selected  when  the  programmer  variables  were  also  included. 
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In  general  then  it  can  be  stated  that  the  three  programmer 
variables  were  of  little  value  for  predicting  errors.  Among 
the  possible  reasons  for  this  finding  is  that  the  RAT  and 
WKLD  measures  as  constructed  to  evaluate  programmers  do  not 
truly  reflect  performance  or  load,  or  that  these  variables 
should  not  be  analyzed  in  a linear  model. 

Clearly,  the  RAT  and  WKLD  measures  as  constructed  and 
applied  are  subjective  measures  used  by  the  program  managers 
to  evaluate  programmers.  Additionally,  the  measures  used  as 
observation  values  in  the  sample  T data  represent  averages 
of  RAT  and  WKLD  for  the  total  number  of  programmers  (anywhere 
from  1 to  15)  responsible  for  programming  each  program  or 
function.  Moreover,  for  each  subsystem  the  correlations  of 
each  of  the  RAT,  WKLD,  and  RAT /WKLD  variables  with  errors 
were  generally  found  to  be  low  to  moderate.  A combination 
of  these  factors  is  contributing  to  the  relative  ineffec- 
tiveness of  these  variables  in  the  prediction  of  errors.  It 
is  of  value  that  these  variables  were  included  in  this  analy- 
sis, in  spite  of  their  insignficant  contributions  to  the  pre- 
diction of  errors.  Since  programmers  do  contribute  to  errors 
(as  a matter  of  fact  they  made  the  errors  analyzed  in  this 
study) , it  is  suggested  that  more  objective  and  standardized 
personnel  and  job  assessment  instruments  be  developed  and 
applied  for  future  studies.  Also,  a non-linear  model  may  be 
more  useful  than  a linear  one  to  evaluate  the  effectiveness 
of  programmer  variables  for  error  prediction  purposes.  Some 
additional  discussion  of  the  programmer  RAT  and  WKLD  variables 
is  presented  in  Section  7.1. 
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With  respect  to  the  consistency  among  predictors  being 
selected  for  each  subsystem  in4,the  error  prediction  equations 
for  errors  predicted  using  the  program  structure  variables 
only  (Table  6-33) , Computational  Statements  (COMP)  and  Comment 
Statements  (COM)  appeared  in  five  of  the  eight  subsystem 
equations.  These  same  predictors  appeared  in  four  of  the  six 
subsystem  equations  of  Table  6-31,  errors  predicted  using 
both  the  program  structure  and  programmer  variables. 

Finally,  regarding  the  effects  on  the  predictability  of 

errors  when  zero  errors  are  delated  (Tables  6-32  and  6-34) , 

2 

there  are  no  significant  changes  in  values  of  R over  most 

subsystems  regardless  of  the  predictors  that  were  being  used 

in  the  analysis.  The  only  exception  is  subsystem  F v/hich 

has  15  (approximately  4155  of  the  total)  zero  error  programs. 

2 2 

The  resulting  increase  was  from  R ■ .80  to  R - .85, 

(Tables  6-33  and  6-34) . However,  as  in  the  results  for 
sample  S,  when  zero  error  programs  are  deleted,  different 
prediction  equations  usually  result,  A good  example  of  this 
can  be  seen  for  subsystem  D in  Tables  6-31  and  6-32.  The 
results  are  presented  here  for  ease  of  comparison. 


All  Observations  Used 
(H»15) 

Zero  Errors 
(H-14) 

Deleted 

IF 

.444* 

LL 

.298 

SYS 

.281 

IF/TS 

.459 

COM 

-.192 

IO/TS 

-.552 

IO/TS 

-.656 

EX/TS 

.678* 

EX/TS 

.555 

COM/TS 

-.206 

R2  - 

.973 

R2  ■= 

.974 

♦Best  single  predictor  of  variables  in  equation. 
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These  differences  in  predictors  and  coefficients  demonstrate 
the  sensitivity  of  multiple  linear  regression  analysis  to 
small  changes  in  correlations. 

The  analysis  of  variance  and  prediction  equation  valida- 
tion results  are  presented  in  Tables  6-37  and  6-38,  respectively, 
for  regressions  where  only  the  program  structure  variables 
are  considered.  The  analysis  of  variance  results  indicate 
that  the  predictor  variables  account  for  a significant  pro- 
portion of  the  total  variance  of  errors/program.  The  F 
statistic  for  each  regression  equation  i3  statistically  signi- 
ficant at  less  than  the  .001  level  of  significance.  For  the 
validation  results  of  Table  6-38,  when  10  programs  are  considered 

as  a validation  sample,  only  a low  to  moderate  shrinkage  in 

2 

the  squared  multiple  correlation  coefficient  (R  ) for  each 
subsystem  occurs.  This  snrinkage  was  the-  greatest  for  sub- 
system A with  the  largest  number  of  observations  (n»51)  . When 
zero  errors  are  deleted  from  subsystem  A (see  Table  6-30) , 

it  is  observed  that  the  shrinkage  is  considerably  less  as 

2 

compared  to  the  shrinkage  of  R when  all  observations  are  U3ed. 

As  is  indicated  in  both  sets  of  validation  results,  all 

equations  showed  increasing  consistency  of  prediction  with  the 

2 

original  values  of  P.  as  larger  validation  samples  were  used 
to  compute  the  shrinkage  estimates  for  each  subsystem.  Con- 
sidering these  results,  it  appears  that  had  actual  validation 
samples  been  available,  each  of  the  obtained  prediction  equa- 
tions would  have  shown  consistent  predictability  of  errors 
in  these  samples . 

In  summary  for  errors/program,  we  can  state  that  high 
predictability  results  when  using  a linear  combination  of 
program  structure  variables  for  errors  collected  throughout  the 
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TABLE  6-37.  ANALYSIS  OF  VARIANCE  TABLES,  SAMPLE  T SUBSYSTEMS 
ERRORS /PROGRAM  = fCPROGRAM  STRUCTURE  VARIABLES), 
ALL  OBSERVATIONS  USED 
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TABLE  6-38.  SAMPLE  T,  VALIDATION  RESULTS  FOR  FIVE  PREDICTOR  REGRESSION  EQUATIONS, 
ERRORS/PROGRAM  = £( PROGRAM  STRUCTURE  VARIABLES),  ALL  OBSERVATIONS  USED 
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[ validation,  integration,  acceptance,  and  operational  phases 

of  software  development.  Predictors  which  heavily  reflect 
jj!  length  of  program  are  generally  found  to  be  the  best  predictors 

of  errors;  however,  other  program  structure-complexity  vari- 
| ables  also  contribute  significantly  to  the  predictability  of 

>;  errors.  And  finally,  the  programmer  rating  and  workload 

j:  variables  as  defined  for  this  study  are  found  to  be  of  little 

value  for  contributing  to  the  prediction  of  errors. 

i,  • 

[ ' 

i : 6.4,2  Results  for  Error  Rate/Program 

I 

The  results  for  the  analysis  of  error  rate/program  presented 
in  this  section  are  as  follows; 


Results 

Tables 

Error  Rate/Program  ■ 

f (Program  Structure  + 

6-40a, 

6-41b 

Programmer  Variables)0 

Error  Rate/Program  - 

f (Program  Structure 

6-42a, 

6-43b 

Variables  Only)5* 

Five  Predictor  Summary 

6-44 

Analysis  of  Variance 

Tables  (Program 

6-45 

Structure  Variables  Only) 

&A11  observations  used 

U 

Zero  error  rates  deleted 

CA  maximum  of  23  predictors  were  available  for  selection 
in  these  predictions. 

J 

A maximum  of  20  predictors  were  available  for  selection 
in  these  predictions. 
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'only  three  predictors  selected  as  a maximum  set  for  this  obtained  r value. 


TABLE  B-1*1*.  FIVE  PREDICTOR  SUMMARY,  ERROR  RATE/PROGRAft  (CONTINUED} 


K OU  H ^ O' 

k..  Pn  r4  ITi  ^ J 

a m n j n ® N 

® * * * 1 • • 


***  CJ  Vfl  N N J "tt 

X m,  "<  4-  CS  v£  JT 

UJ  • • • * * * 


**  rs 

N (S  N ^ 

'a.  jf  »■«  *a  >a 


^ h»« 

O'  J «*! 

m k w> 


£ 

n rs  k/>  ^ 

^ k\  *a 

j r«>  ^ » 


J u5  K 

pH  A.  o 

J*  00  r*. 


r«k  rv  ^ ^ 

^ 00  CO  CT'  cfl 

^ 50  h*  O'  O' 


W cG  K. 

u r»  ^ 

kA  ff"»  CO 


kA  f*k  t<k 

LA  N A 05 

kA  y M O ^ 


kA  <N  <S 

IA  9 H 

<N  rn  kO 


k J iA  iA  J 

mH  00  •“•  »A  ^ 

PA  A*  v£  (T*  00 


- <S  Jf 

hn  *?4  i-H 

kfl  fl>  » 


H N ^ « H 

S.  -T  M 00  kO 

stf  ^ 50  ^ O' 


CO  00 

PA  *M  JO 

CS  O *A 


m lO  rs  05 

^ pk  la  ^ ^ 

>A  ■£  4*  N,  Vfl 


UJ 

UJ 

UJ 

UJ 

J- 

UJ 

H 

h* 

UJ 

< 

K 

4 

< 

H- 

< 

u 

a t 

« 

tf 

OL 

0L 

« 

□C 

< 

OL 

OL 

OL 

O 

UJ 

cn 

O 

01 

o 

o 

OL 

O 

CL 

O 

0L 

0L 

O 

u 

LO 

CL 

<* 

0L 

OL 

OL 

OL 

UJ 

CL 

UJ 

i/) 

UJ 

OL 

UJ 

A 

UJ 

A 

h- 

a 

LU 

k 

wn 

h- 

- :i 

SLCN  fN 

k 

;< 

TS 

5ICs| 

” N 

MM 

K 

u 

U 

U CL  JC 

i«  u» 

U 

U 

H CL  CL 

> 

0L  O IT< 

D a:  uj 

►-  u t- 

u n < 


i/l  lA  c w 

-J  c:  -J 

• a (.*  w 


C-94 


; >*P0~-  - ■ - 4/’ 


* V V 


-BEST  SINGLE  PREDICTOR 

a0NLY  FOUR  PREDICTORS  SELECTED  AS  A MAXIMUM  SET  FOR  THIS  R VALUE. 
fa0NLY  THREE  PREDICTORS  SELECTED  AS  A MAXIMUM  SET  FOR  THIS  R VALUE 


A list  of  sample  T predictor  variables  that  were  candidates 
for  entry  in  the  prediction  equations  for  error  rate  is  pre- 
sented in  Table  C-3  of  Appendix  C. 

The  summary  results  for  error  rate  (Table  6-44'  indicate 
that  regardless  of  whether  or  not  the  programmer  variables 
have  been  made  available  for  selection  in  the  prediction 
equation,  moderate  to  high  predictability  is  achieved  for 
each  subsystem  of  sample  T,  with  the  exception  of  subsystem  F. 
The  best  single  predictor  of  error  rate  is  one  of  the  two  nor- 
malized Program  Interface  variables,  AP/TS  and  SYS/TS,  which 
appear  consistently  as  the  best  predictions  in  seven  of  the 
eight  subsystems  when  the  program  structure  variables  with 
zero  error  rates  deleted  are  considered.  Additionally,  these 
interface  variables  account  for  a large  portion  of  the  overall 
predictability  in  each  subsystem,  and  other  program  complexity 
variables  appear  to  be  significantly  contributing  to  error 

rate  predictions  (i.e.,  as  evidenced  by  the  difference  between 
2 2 

R and  r * ^ error  rate  for  eac1'  subsystem). 

Also  a consistent  finding  in  the  analysis  is  that  the 
majority  of  predictors  in  the  equations  for  error  rate  are 
the  normalized  variables.  This  finding  is  observed  for  all 
prediction  results  (Tables  6-4Q  to  6-43)  obtained  for  error 
rate . 

By  examining  Table  6-40,  it  is  noted  that  when  the  pro- 
grammer variables  RAT,  WKLD , and  RAT/WKLD  are  made  available 
for  selection,  only  the  Programmer  Rating  variable  (RAT)  is 
selected,  and  then  only  for  two  of  the  six  subsystems.  In 
Table  6-42  which  reports  the  predictions  after  removing  the 
programmer  variables  from  tne  analysis,  the  results  show  that 
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(1)  the  predictability  of  error  rate  is  essentially  unaffected, 
and  {2)  the  predictors  selected  for  each  subsystem  do  not 
significantly  differ  from  those  selected  when  the  programmer 
variables  are  made  available  to  enter  the  regression  equation. 
In  the  analysis  the  correlations  of  each  of  the  programmer 
variables  with  error  rate  were  observed  to  be  low  over  each 
of  the  subsystems;  low  to  moderate  correlations  were  observed 
for  these  variables  with  the  errors/program  measure.  Thus, 
the  programmer  variables  as  defined  for  this  study  have  a 
negligible  contribution  to  the  predictability  of  both  errors 
per  program  and  error  rate  per  program. 

To  examine  the  consistency  of  predictors  selected  over 
all  subsystems,  all  results  (Tables  6-40  thru  6-43)  show  that 
the  three  normalized  variables  AP/TS,  SYS/TS,  and  EX/TS,  are 
the  variables  most  frequently  appearing  when  predicting  error 
rate  over  all  23  regression  equations. 

Examining  the  predictability  of  error  rate  when  zero 

error  rates  are  deleted,  it  is  found  that  with  the  exception 

2 

of  subsystems  F and  G,  no  major  changes  in  the  value  of  R 
over  each  of  the  subsystems  are  observed.  Since  subsystems  F 
and  G are  quite  unique  relative  to  each  other  and  to  the  six 
remaining  subsystems  with  respect  to  the  large  percentage  of 
error  free  programs  in  each,  a further  analysis  of  error  rate 
in  F and  G is  presented  in  Section  7.2, 

The  analysis  of  variance  results  presented  in  Table  6-45 
indicate  that  the  F statistic  for  each  subsystem's  regression 
equation  is  significant  at  less  than  the  .001  level  of  signi- 
ficance. The  linear  predictions  then  account  for  a statisti,- 
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cally  significant  proportion  of  the  variability  in  error  rate, 

2 

wherein  this  proportion  is  measured  by  the  values  of  R for 
each  subsystem. 

6.4.3  Sample  T Prediction  Consistency  Analysis 

The  predictions  for  both  errors  and  error  rate  are 
generally  moderate  to  very  high  for  each  data  sample  over  all 
predictions  obtained.  For  the  analysis  of  sample  T results, 
a set  of  five  predictor  variables  is  used  uniformly  across 
all  subsystems  to  predict  errors,  and  a different  cat  of 
five  is  used  to  predict  error  rate.  A significant  change  in 
the  squared  multiple  correlation  coefficient  from  that 
obtained  for  the  bast  five  predictors  in  each  subsystem  would 
indicate  inconsistency. 

For  errors/program  the  five  predictors  from  subsystem  A 
are  used  (i.e,,  TS,  AP,  I/O,  COMP,  and  COM).  For  error  rate, 
the  subsystem  A predictors  (SYS,  AP/TS,  SYS/TS , EX/TS,  COM/TS) 
are  also  chosen.  These  predictors  were  used  since  they  were 
automatically  selected  by  the  regression  procedure  based  on 
the  largest  sample  of  observations  (n=51)  available  for 
sample  T. 

The  results  of  this  consistency  analysis  are  presented 
in  Tables  6-46  (for  errors)  and  6-47  (for  error  rate).  All 
predictors  in  each  set  of  variables  are  forced  into  the  equa- 
tion in  order  to  obtain  comparable  results  over  each  subsystem. 
Since  the  predictors  for  subsystem  A had  not  been  selected 

by  the  regression  procedure  for  the  other  subsystems,  a 

2 

reduction  in  the  value  of  R for  each  subsystem  except  A 
was  expected. 
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For  errors/program,  it  ia  observed  that,  using  the  same 
set  of  predictors,  moderate  to  high  predictability  is  maintained 
over  all  subsystems.  For  error  rate,  less  consistency  of 
prediction  results.  Nevertheless,  for  six  of  the  eight 
subsystems  (A  thru  F)  error  rate  predictions  are  still  in 
the  moderate  to  high  range. 

one  particular  interpretation  of  these  results  is  that 
since  the  same  variables  appear  in  the  equations  for  each 
subsystem,  an  estimate  of  how  these  variables  contribute  to 
errors  in  general  may  be  obtained.  In  spite  of  the  apparent 
differences  among  the  subsystems  of  sample  T,  there  is  re- 
markable consistency  in  results  of  applying  the  same  set  of 
five  predictors  to  all  subsystems.  These  results  may  then 
apply  to  programming  in  general,  or  at  least  to  command  and 
control  systems  using  JOVIAL  J4. 

Since  the  predictability  of  errors  and  error  rate  was 

higher  for  sample  T programs  than  for  those  of  sample  S, 

predictions  for  errors  and  error  rate  were  obtained  using 

the  two  distinct  sets  of  predictors  over  all  24?  program 

observations  of  sample  T.  These  results  are  reported  in 

Table  6-48.  The  predictions  were  observed  to  be  in  the  moderate 

range  for  both  errors  and  error  rate.  Given  the  larger  sample 

involved  and  the  increased  variability  over  all  variables 

that  results  from  this  aggregation  over  all  subsystems,  the 

predictions  may  be  generally  moro  indicative  of  the  true 
2 

values  of  R,  R , and  other  correlation  and  regression  statis- 
tics in  the  population  of  programs  of  which  these  249  programs 
are  but  a sample. 


TABLE  6-48.  SAMPLE  T PREDICTION  RESULTS  USING  ALL  SUBSYSTEMS 
(N=249) 


VARIABLE 

REGRESSION 

COEFFICIENT 

PREDICTION 

SUMMARY 

ERRORS/PROGRAM 

TS 

. 75  8" 

2!!,ERRORS=  .765 

\ A P 

.218 

r !!,  ERRORS-  .586 

; i/o 

- .050 

Rs  .797 

r COMP 

- .116 

R2=  .635 

COM 

.043 

STD . ERR.  EST.s  6.464 

; 

, 

ERROR  RATE/PROGRAM 

i 

SYS 

- .041 

2:c,ERRORS=  .641 

AP/TS 

. 459" 

r «,ERR0RS=  .412 

SYS/TS 

.206 

R=  .714 

EX/Tj 

.235 

R2=  .510 

COM/TS 

.085 

STD.  ERR.  EST.=  2.171 

:!  PEST  SINGLE  PREDICTOR 


Several  comments  are  important  here  regarding  the 
consistency  analysis  results  obtained  using  each  subsystem 
(Tables  6-46  and  6-47)  vis-a-vis  the  results  observed  using 
all  sample  T programs  (Table  6-48) . First,  for  errors/program, 
at  the  subsystem  level  (Table  6-46) , Total  Source  Instructions 
(TS)  is  the  best  predictor  of  errors,  and  the  four  program 
complexity  variables  (AP,  I/O,  COMP,  COM)  are  contributing 
significantly  to  error  predictability.  In  contrast,  at  the 
aggregate  level  over  all  subsystems.  Total  Source  Instructions 
alone  accounts  for  almost  92S  of  the  total  variance  explained 
by  the  five  predictors.  At  this  level  the  four  complexity 
variable  have  only  a negligible  effect  on  the  predictability 
of  errors/program.  An  important  implication  is  suggested  by 
these  results.  When  estimating  the  total  number  of  errors 
likely  to  be  found  as  a result  of  formal  testing  for  a group 
of  programs  that  are  functionally  heterogeneous  (and  similar 
in  nature  to  those  of  sample  T) , the  size  or  length  of  each 
program  may  be  the  single  most  important  predictor.  Whereas 
for  programs  that  are  more  functionally  homogeneous,  other 
program  complexity  variables  in  addition  to  program  length 
should  be  considered  in  the  prediction  process  to  achieve 
some  initial  estimate  of  errors/program. 

For  error  rate/program,  regardless  of  whether  the  pre- 
diction is  based  on  sets  of  programs  at  the  individual  subsystem 
level  or  using  the  entire  sample  of  249  programs,  each  of 
the  program  complexity  variables  (SYS,  AP/TS , SYS/TS , EX/TS, 
COM/TS)  in  combination  are  contributing  to  the  predictability 
obtained.  At  the  subsystem  .level  this  is  true  for  at  least 
five  of  the  eight  subsystems.  A,  C,  E,  G and  H.  This  result 
suggests  that  the  overall  variability  of  the  error  rate  measure 
and  its  predictors  (four  of  which  were  normalized  program 
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complexity  measures}  were  essentially  unaffected  by  the 
aggregation  over  all  subsystems.  Thus,  error  rate/program 
and  its  predictors  may  be  more  stable  measures  to  be  used  for 
prediction  purposes,  regardless  of  the  functional  mixture  of 
programs  being  considered. 
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ADDITIONAL  ANALYSIS 


7.1  Error  Rate  and  Programmer  Variables 

Since  the  programmer  Rating  (RAT)  and  Workload  (WKLD) 
variables  proved  to  be  of  no  predictive  value  to  errors  or 
error  rate  when  combined  with  program  structure  variables, 
it  was  decided  that  a more  thorough  analysis  of  these  vari- 
ables particularly  as  they  related  to  error  rate,  would  be 
performed. 

Essentially  in  this  analysis  the  average  workload  and 
average  error  rate  for  different  categories  of  programmer  ratings 
are  calculated.  Then,  how  different  levels  of  programmer 
rating  and  workload  affect  the  error  rate  is  determined.  This 
analysis  is  performed  over  all  program-observations  of  sub- 
systems A thru  E.  The  results  of  this  analysis  are  presented 
in  Table  7-1. 

Each  of  the  average  error  rates  is  statistically  tested 
(using  the  * t ' test)  for  significant  differences  with  each 
neighboring  mean  value  in  the  following  manner:  the  average 

error  rate  for  programmers  rated  less  than  10  (i.e.,  1.50) 
was  tested  and  found  not  statistically  different  from  the 
average  error  rate  of  programmer's  with  ratings  from  11  to 
12  (i.e.,  3 . 2'2 ) . The  average  error  rates  for  these  two  groups 
combined  was  then  tested  for  a significant  difference  with 
the  next  error  rate  value  (1.62),  showing  again  no  significant 
difference.  The  average  of  the  three  error  rates  was  then 
compared  with  the  value  of  2.99;  again  no  statistical  signifi- 
cance was  found.  (This  same  procedure  was  followed  for  each 
of  the  remaining  groups  of  programmer  ratings) . The  only 
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TABLE  7-1.  SAMPLE  T,  ERROR  RATE  AND  PROGRAMMER 
VARIABLE  RELATIONSHIPS 


PROGRAMMER 

RATING 

NUMBER  AVERAGE 

PROGRAMS  WORK  LOAD 

AVERAGE 
ERROR  RATE 

< 10 

8 

1.00 

1.50  ~ 

11-12 

18 

.97 

3.22 

2.43 

13-14 

15 

1.10 

1.62 

15-16 

22 

1.18 

2.99  _ 

17 

27 

1.33 

4.36  J 4. 36 

13 

16 

1.33 

2.05 

19 

19 

1.34 

2.56 

2.44 

20 

10 

1.34 

2.85 

CN= 1 3 5 ) 
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statistically  significant  differences  that  exist  occur  as 
indicated  between  the  bracketed  groupings  of  means;  i.e., 

(1)  between  the  average  error  rates  for  programmers  rated  in 
the  <.10  to  16  and  17  categories,  and  (2)  between  the  error 
rates  for  programmers  rated  17  or  higher  (i.e,,  the  difference 
between  4.36  and  2.44  was  significant  at  the  .05  level). 

Basically  the  following  observations  can  be  made  from 
these  results: 

(1)  the  lower  rated  programmers  with  lighter  workloads 
do  produce  significantly  fewer  errors  per  100  lines 
of  code  as  compared  with  the  higher  rated  programmers 
who  had  the  heavier  workloads; 

(2)  the  same  high  rated  programmers  (i.e.,  RAT  ■ 17) 
produce  significantly  more  errors  per  100  lines 

of  code  as  compared  with  the  top  rated  programmers, 
irregardless  of  the  relatively  high  workload  each 
group  had;  and 

(3)  the  highest  rated  programmers  having  the  heaviest 
workload  produced  as  many  errors  per  100  lines  of 
code  as  did  the  lowest  rated  programmers  having 
the  lowest  workload. 

Additionally,  using  this  kind  of  analysis  as  compared  to  the 
linear  regression  approach,  one  can  clearly  see  the  non-linearity 
in  the  relationship  of  error  rate  with  both  programmer  workload 
and  rating.  That  is,  as  both  RAT  and  WKLD  variables  increase, 
error  rate  also  increases  up  to  a point  (i.e.,  RAT  * 17  ) and 
then  becomes  smaller  with  still  increasing  values  of  RAT  and  WKLD 


As  cited  in  information  provided  about  the  RAT  and  WKLD 
variables  of  sample  T,  many  of  the  programmers  specifically 
rated  in  the  RAT  ■ 17  category  were  managers  not  only  managing 
the  software  development  effort  but  also  programming  at  the 
same  time.  This  seems  to  suggest  that  managers  who  also 
contribute  to  the  programming  effort  contribute  significantly 
to  errors  in  programs , more  so  than  programmers  who  do  nothing 
but  program. 

The  method  of  analysis  used  here  best  approximates  the 
standard  analysis  of  variance  approach  typically  used  to 
analyze  experiments  involving  one  or  more  factors.  Although 
the  analysis  of  variance  methodology  is  generally  not  used 
for  purposes  of  prediction,  when  properly  annlied  it  can  be 
very  useful  for  identifying  relationships  among  data  variables 
that  may  go  undiscovered  using  linear  regression  analysis. 

Additionally,  the  kinds  of  interpretations  of  data 
relationships  that  can  be  made  using  the  analysis  of  variance 
approach  may  in  many  instances  have  more  operational  meaning 
than  those  allowed  using  the  regression  approach.  For  this 
reason,  the  analysis  of  variance  approach  and  other  methods 
of  analysis  (e.g.,  contingency  table  analysis  using  chi-square 
tests,  non-linear  regression  models)  are  strongly  suggested 
as  additional  methodologies  which  can  and  should  be  employed, 
where  applicable,  in  future  software  reliability  analysis 
studies  where  programmer,  project,  test,  software  environment, 
and  error  data  are  all  available  for  analysis. 
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7.2  Error  Rate  and  Source  Instructions 


Throughout  the  study  of  error  rate/program  it  was  observed 
that  the  correlation  between  error  rate  and  total  source 
instructions  was  consistently  low,  but  negative,  with  one 
exception.  For  each  of  the  sample  T subsystems,  the  correla- 
tions, as  reported  in  Table  7-2,  exist  between  error  rate  and 
the  Total  Source  Instructions  variable  (TS) , when  the  error 
free  programs  are  included  and  then  excluded  from  the  analysis. 

Basically,  the  consistent  increase  in  magnitude  of  each 
of  these  correlations  (with  the  exception  of  subsystem  G) 
and  the  consistency  of  the  low  to  moderate  linear  relationship 
between  error  rate  and  source  instructions  over  all  subsystems, 
when  the  error  free  programs  were  deleted,  supports  the  hypo- 
thesis that  longer  programs  are  less  thoroughly  tested.  The 
hypothesis  asserts  that  longer  programs  were  less  thoroughly 
tested  relative  to  the  shorter  programs.  Thi3  is  likely  to  be 
the  case  since  as  the  length  of  a program  increases,  a more  rapid 
than  linear  rate  of  increase  in  the  number  of  paths  through  the 
program  usually  would  occur,  thus  increasing  a program's  com- 
plexity. This  increased  complexity  would  then  result  in  t.ie 
longer  programs  requiring  more  time  to  test  and  thus  they  might 
very  well  be  less  thoroughly  tested. 

The  fact  that  this  hypothesis  can  be  considered  as  a 
plausible  explanation  here  and  that  these  results  are  not  just 
correlational  anomalies  is  borne  out  in  the  following  analysis 
of  error  rate  presented  for  both  subsystems  F and  G (see 
Tables  7-3  and  7-4,  respectively). 
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TABLE  7-2. 


CORRELATIONS  BETWEEN  ERROR  RATE  AND  TOTAL  SOURCE 
INSTRUCTIONS  FOR  SAMPLE  T SUBSYSTEMS 


subsystem 

rTS, ERROR  RATE 

USING  ALL 

WHEN  ERROR  FREE 

OBSERVATIONS 

PROGRAMS  DELETED 

t- 


A 

-.238 

C51) 

-.361 

(44) 

B3 

-.671 

(16) 

-.671 

(16) 

C 

-.235 

(39) 

-.317 

(36) 

D 

-.508 

(15) 

-.555 

(14) 

E 

-.414 

(14) 

-.496 

(13) 

F 

-.096 

(37) 

CM 

-3- 

-T 

0 

01 
CM 

G 

.167 

(45) 

-.070 

(34) 

H 

-.211 

(32) 

-.251 

(31) 

Subsystem  b had  no  error  free  programs. 
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TABLE  7-3.  SUBSYSTEM  F ERROR  RATE  ANALYSIS 
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For  both  sets  of  results,  all  programs  were  classified 
as  being  either  above  or  below  the  average  value  of  source 
instructions;  the  average  value  of  error  rate/program  was 
then  computed  for  each  classification.  Essentially  for  both 
subsystems,  no  differences  were  observed  between  the  average 
error  rates  for  the  shorter  programs  versus  the  longer 
programs,  whereas  the  differences  between  the  average  length 
of  the  shorter  compared  to  that  of  the  longer  programs  was 
strikingly  different.  Now,  when  the  error  free  programs  are 
removed  and  only  the  programs  with  errors  are  classfied 
according  to  the  same  procedure,  significant  differences 
between  the  error  rates  for  the  shorter  versus  the  longer 
programs  do  exist.  Finally,  when  this  same  set  of  programs 
is  classified  even  further  into  three  categories  as 
indicated,  the  negative,  low  level  linear  relationship  between 
error  rate  and  source  instructions  becomes  apparent. 

These  results  are  interpreted  as  lending  support  to  the 
hypothesis  that  longer  programs,  in  particular  those  with  no 
errors  reported,  were  less  thoroughly  tested  than  the  shorter 
ones . 

In  summary  then,  it  is  not  known  whether  the  programs  with 
zero  reported  errors  are  truly  error  free.  When  these  programs 
are  considered  to  have  unreported  or  latent  errors  '•emaining 
in  them  and  are  removed  from  the  analysis,  then  the  consistency 
of  the  relationship  between  error  rate  and  length  of  program 
becomes  more  strikingly  apparent,  and  hypotheses  concerned  with 
thoroughness  of  testing  of  the  longer  versus  that  of  the  shorter 
programs  become  more  readily  testable. 
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8.0 


CONCLUSIONS  AND  RECOMMENDATIONS 


8.1  Conclusions 

The  major  purpose  of  this  study  was  to  determine  how 
predictable  programming  error  measures  are  from  a combination 
of  program  characteristic  variables  using  multiple  linear 
regression  analysis.  By  examining  the  degree  of  predictability 
obtained,  the  effectiveness  of  the  linear  regression  model 
in  software  error  prediction  studies  may  then  be  evaluated. 

With  respect  to  this  purpose,  given  the  analysis  results 
obtained,  the  following  conclusions  can  be  drawn: 

• The  predictability  of  programming  error  measurements 
are  variable,  ranging  from  very  low  to  very  high. 

For  the  errors/program  measure,  predictability  is 
found  to  be  consistently  in  the  moderate  to  very 
high  range.  For  the  error  rate/program  measure, 
predictability  is  generally  less  than  that  obtained 
for  errors/programs j with  the  predictability  ranging 
from  very  low  to  high  and  with  less  consistency  than 
errors/program  throughout  all  the  predictions  obtained. 

• The  variability  in  the  predictions  obtainod  over 
both  data  samples  is  considered  to  be  strongly 
related  in  varying  degrees  to  each  of  the  following 
factors : 

a.  functional  differenc . among  the  various 
programs  that  were  ueveloped 

b.  differences  in  the  programming  language  used 
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c.  the  length  of  time  formal  error  data  collection 
was  carried  out 

d.  the  amount  and  thoroughness  of  testing  of 
each  program 

e.  inadequacy  of  the  linear  model  to  provide 
perfect  predictability 

f.  other  programmer,  project,  and  management 
factors  affecting  the  software  development 
process . 

8.2  # Direct  Recommendations 


The  following  set  of  recommendations  discuss  measures 
that  can  be  taken  to  bring  about  an  increased  consistency  of 
prediction  of  programming  error  measures  in  future  software 
error  prediction  studies.  These  recommendations  pertain  to 
the  predictor  variables,  the  programming  error  variables, 
applications  of  the  multiple  regression  procedure,  and  software 
testing  procedures. 

(1)  Predictor  Variables  - Predictor  variables  should  be 
accurately  identified  and  concisely  defined  prior  to  the  begin- 
ning of  software  development.  Predictors  should  be  identified 
which  not  only  reflect;  selected  program  and  language-specific 
characteristics,  but  moreover  they  should  include  a variety 
of  candidate  programmer,  management,  and  software  development 
environment  variables  which  are  suspected  of  reasonably 
affecting  the  quality  and  reliability  of  the  software  being 
developed . 
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A baseline  set  of  predictor  variables  should  be  defined 
and  applied  over  the  general  range  of  software  projects  so 
that  consistency  of  measurement  can  be  obtained.  This  baseline 
set  can  be  compiled  from  the  results  of  this  study's  five  and 
ten  predictor  summaries.  As  further  studies  identify  addi- 
tional predictor  variables,  the  baseline  set  should  be 
expanded. 

Data  for  these  predictor  variables  should  be  collected 
throughout  the  successive  phases  of  software  development.  Any 
significant  changes  or  modifications  should  be  recorded, 
dated,  and  the  cause  of  the  error  determined. 

In  order  to  benefit  the  generalizability  of  future  error 
prediction  studies  wherein  different  programming  languages 
will  be  involved,  present  and  future  research  effort  should 
be  directed  toward  identifying  those  language  and  program  cnarac- 
teristics  that  may  be  equated  or  made  comparable  between  two 
or  more  languages . 

(2)  Programming  Error  Measures  - These  measures  should 
be  concisely  defined  and  collected  throughout  all  phases  of 
software  development.  In  addition  to  a description  of  the 
errors,  their  symptoms,  and  the  program  changes  required  to 
correct  the  errors  / other  data  should  be  collected  which  would 
include  when  the  error  was  found,  the  method  used  to  detect 
the  error,  and  an  estimate  of  the  total  effort  (e.g.,  man-hours, 
computer  time,  documentation  changes,  etc.)  involved  in  error 
identification  and  correction. 
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Of  particular  importance  for  error  prediction  purposes 
is  that  a taxonomy  or  typology  of  programming  eirors  be  developed 
such  that  more  definitive  predictions  can  be  developed  for 
separate  versus  a gross  aggregation  of  error  types. 

Each  of  the  errors,  regardless  of  error  type,  should  further 
be  classified  and  weighted  with  respect  to  their  criticality 
or  severity  for  impeding  the  achievement  of  the  software  develOD- 
ment  project  objectives.  Once  this  has  been  accomplished,  then 
predictions  of  errors  having  different  criticality  can  be  per- 
formed and  the  most  important  variables  for  each  can  then  be 
identified.  Existing  error  collection  tools  could  be  expanded 
to  classify  and  weight  errors.  Manually  collected  error  data 
should  also  be  classified  and  weighted  either  manually  or  by 
interfacing  with  error  collection  tools. 

(3)  Multiple  Regression  Applications  - A parallel  pre- 
diction approach  which  utilizes  multiple  regression  analysis 
applied  at  various  milestones  or  stages  of  software  development, 
testing,  and  operational  usage  is  recommended  for  future  error 
analysis  and  prediction  studies. 

The  parallel  prediction  approach  proposes  to  make  separate 
predictions  of  errors  during  each  of  the  designated  stages 
during  the  project,  using  a specified  set  of  predictors  for 
which  data  would  be  collected  at  each  of  these  stages.  Of 
special  interest  in  this  analysis  is  the  identification  of 
particular  criterion  error  measures  (e.g.,  gross  error  counts 
or  error  rates  for  given  error  types,  or  errors  weighted  by 
severity)  whose  overall  predictability  is  changing  meaningfully 
ever  time.  Also  of  interest  at  each  stage  is  the  relative 
importance  of  each  of  the  predictor  variables  in  the  equations. 
Clearly,  any  of  the  predictors  found  to  systematically  increase 
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(or  decrease)  in  explanatory  power  over  time  is  deserving  of 
further  attention. 

Additionally,  since  data  would  be  available  from  prior 
time  periods  using  this  approach,  one  could  investigate  any 
time  lag  relationships  that  may  exist  between  the  predictors 
and  errors  when  predicting  errors  at  later  stages  in  the 
project. 

Using  this  parallel  predictions  approach,  both  linear 
and  non-linear  models  should  be  investigated. 

(4)  Software  Testing  Procedures  - Throughout  this  report 
the  thoroughness  of  testing  in  each  of  the  two  data  samples 
was  repeatedly  stressed  as  an  important  factor  contributing 
towards  the  identification  of  errors  in  programs.  It  is 
strongly  recommended  that  the  amount  of  testing  of  program 
modules  be  measured  in  software  development  projects  as  much 
as  possible  so  that  errors  weighted  by  amount  of  testing  can 
be  analyzed.  This  measure  of  amount  of  testing  should  address 
possible  paths  tested  and  range  of  possible  inputs  and  outputs 
exercised . 
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8.3  Recommendations  for  Further  Research 


Throughout  the  course  of  this  study,  the  need  to  perform 
additional  research  on  the  available  data  became  obvious. 
However,  such  research  was  beyond  the  scope  of  this  contract. 
Four  of  the  major  topics  of  interest  and  value  to  other  software 
quality  and  reliability  studies  are  presented. 

(1)  Non-Linear  Regression  Analysis  - This  study  suggests 
that  non-linear  regression  will  improve  both  the  consistency 
of  the  predictions  and  the  predictability  of  each  of  the  pro- 
gramming error  measurements.  A continued  analysis  of  both 
sample  S and  sample  T data  along  with  other  available  data 
samples  should  be  investigated  for  error  prediction  purposes 
using  non-linear  regression  models. 

Consider,  for  example,  that  the  actual  error  rate  of 
programs  that  have  been  thoroughly  tested  increases  up  to  a 
certain  level  for  a given  number  of  source  instructions,  ^.nd 
then  it  increases  only  slightly  thereafter  for  continuing 
increases  in  program  length.  That  is,  the  error  rate  becomes 
almost  constant  after  a certain  program  length.  This  error 
rate  could  be  estimated  for  the  sample  S and  sample  T aata 
using  the  equation 

Y ' = a + bx  + cx^  (8.0) 

The  independent  variable  (x)  in  this  equation  is  the  total 
source  instructions  variable  (XI  or  TS)  for  either  sample. 

The  dependent  or  predicted  variable  (Y*)  is  the  new  estimated 
value  of  error  rate  obtained  for  each  program  in  the  two  data 
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samples  that  are  presently  available.  This  newly  estimated 
or  revised  error  rate  (hypothesized  for  thoroughly  .tested 
programs)  would  replace  the  observed  error  rates  for  many  of 
the  programs  used  in  this  study,  which  were  probably  not 
thoroughly  tested.  Figure  8-1  depicts  the  observed  error 
rate-program  length  relationships  for  the  three  projects  of 
sample  S as  contrasted  with  the  new  or  revised  estimates  of 
error  rate  using  the  non-linear  model  described  above. 

Using  these  revised  or  new  estimates  (Y1)  of  what  the 
actual  error  rates  for  these  sample  programs  should  be,  this 
new  variable  can  be  predicted  using  various  non-linear  forms 
of  the  predictor  variables  in  a multiple  regression  equation. 

(2)  Predictions  By  Error  Tvoe  and  Severity  - As  mentioned 
in  the  earlier  discussion  of  recommendations,  it  is  strongly 
recommended  that  errors  be  classified  according  to  type  and 
severity  and  that  predictions  be  obtained  for  those  classifi- 
cations . During  this  study  no  data  was  available  for  either 
sample  that  would  enable  the  assignment  of  such  classif ications 
to  the  errors  in  each  program.  If  this  data  is  available, 

it  woulc  then  be  possible  to  develop  different  regression 
equations  for  different  types  of  errors.  The  results  of  such 
an  analysis  should  show  higher  predictability  and  give  insights 
more  directly  related  to  cause-effect  than  were  obtained  by 
aggregating  errors . 

(3)  Analysis  of  Error  Free  Programs  - If  the  zero  reported 
error  programs  that  appeared  throughout  the  analysis  were  in 
reality  error  free,  then  an  analysis  directed  at  determining 
the  characteristics  of  these  programs  would  be  very  meaningful. 
Those  characteristics  which  differentiate  error  free  from 
error  prone  modules  could  be  determined. 
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(4)  Analysis  of  Constant  Size  Programs  - In  addition  to 
the  normalization  procedure  used  to  construct  additional  pre- 
dictor variables  for  this  study  and  the  partial  correlation 
procedure  discussed  in  Section  5.0,  one  straightforward  pro- 
cedure for  controlling  for  the  effect  of  program  length  in 
this  analysis  would  be  to  analyze  different  sets  of  constant 
size  programs  in  each  given  sample.  This  could  be  accomplished 
by  developing  groups  of  programs,  for  example,  that  had  an 
average  program  length  of  50,  150,  250,  etc.  source  instructions 
each.  Then  only  tnose  programs  falling  within  1 or  1/2  standard 
deviation  above  and  below  the  mean  would,  for  statistical 
purposes,  be  considered  as  a group  with  constant  program  length. 
A major  benefit  of  this  analysis  would  be  to  identify  how  the 
predictability  of  errors  and  error  rate  Ji ffers  over  each  of 
these  groups , and  what  variables  are  the  most  important 
predictors  of  errors  for  smaller , medium , and  longer  length 


programs . 
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8.4  Proposed  Support  Tools  and  Techniques 

To  most  effectively  apply  the  results  of  the  study  toward 
obtaining  more  error  free  software,  support  tools  and  techniques 
are  required.  This  section  proposes  tools  and  techniques 
conceptualized  throughout  this  study. 

(1)  Collection  of  Data  for  Predictor  Variables  - Predictor 
variables  which  define  program  characteristics  can  be  measured 
from  source  code.  Those  measurement  programs  already  in  exist- 
ence for  specific  projects  are  known  as  scanner  programs.  To 
enable  the  most  accurate  and  effective  method  of  measuring  the 
predictor  variables,  Language  Scanners  should  be  developed  as 
part  of  support  software  packages . Language  Scanners  should 
measure,  at  a minimum,  the  baseline  set  of  predictor  variables 
defined  under  the  first  recommendation  of  Section  8.2.  Language 
Scanners  can  easily  be  added,  for  example,  to  a programming 
support  library  which  stores  and  maintains  source  code  as  well 
as  performs  all  compilations.  A Language  Scanner  can  be  pro- 
vided to  support  each  language  in  the  same  manner  as  a pre- 
compiler is  provided  for  each  structured  language. 

(2)  Evaluation  of  High  Order  Languages  - Existing  high 
order  languages  should  be  analyzed  to  identify  those  charac- 
teristics  which  are  most  closely  correlated  with  errors.  When 
such  characteristics  are  identified,  it  is  necessary  to  determine 
which  of  these  characteristics  do  cause  errors.  For  those  cases 
when  cause  is  established,  preventive  measures  could  be  introduced. 
This  type  of  evaluation  can  apply  not  only  to  existing  languages 
but  also  to  any  language  under  development  or  modification. 
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(3)  Test  Support  Tools  - Test  support  tools  are  under 
development  throughout  the  software  industry,  particularly 
in  the  areas  of  identifying  program  paths.  Such  a tool  should 
also  contain  a capability  for  weighting  (or  accepting  manual 
weighting)  the  various  paths.  Weighting  might  include  such 
factors  as  frequency  of  use  and  criticality.  Testing  emphasis 
could  then  be  directed  according  to  the  weighting  scheme. 

Another  test  support  tool  to  be  developed  is  an  input/ 
output  range  definer.  Representative  inputs  can  be  selected 
for  path  testing  from  the  required  range  of  values.  The 
number  of  different  outputs  can  be  compared  against  the  re- 
quired range  of  outputs.  Untested  outputs  can  be  identified. 

Both  testing  aids  proposed  here  will  assist  in  iden- 
tifying desired  testing  and  in  determining  where  available 
resources  should  be  applied  during  testing.  A combination 
of  number  of  paths  exercised  and  the  range  of  inputs  and 
outputs  tested  is  a further  measure  related  to  software 
system  reliability. 
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t 8.5  Summary  of  Recommendations 

The  following  list  is  a summary  of  recommendations 
resulting  from  this  study.  Items  (1)  through  (4)  are  direct 
applications  of  the  results. 

(1)  Definition  and  collection  of  data  for  predictor 
variables. 

(2)  Error  classification  and  weightings. 

(3)  Apply  regression  models,  both  linear  ana  non-linear, 
throughout  the  software  development  process. 

(4)  Define  testing  techniques  which  measure  thoroughness 
of  testing. 

Items  (5)  through  (9)  recommend  further  research. 

(5)  Investigate  non-linear  multiple  regression. 

(6)  Classifiv  errors  according  to  type  and  severity  and 
obtain  predictions  for  these  classifications. 

(7)  Continue  analysis  of  error-free  programs  from  a 
broader  data  sample. 

(8)  Continue  analysis  of  programs  grouped  by  relative 
size  from  a broader  data  sample. 

(9)  Apply  the  prediction  model  obtained  by  this  study 
over  a broader  data  sample. 
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Items  (10)  through  (12)  identify  software  support  tools  and 
techniques  which  will  assist  in  implementing  the  preceeding 
recommendations . 

(10)  Develop  Language  Scanners  to  measure  predictor 
variables . 

(11)  Evaluation  of  high  order  languages. 

(12)  Develop  test  support  tools  which  a)  identify  and 
weight  program  paths,  and  b)  determine  representative 
test  input  to  produce  required  outputs. 
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appendix  a 

(This  Appendix  contains  a discussion 
of  considerations  on  the  measure  of 
error  to  be  analyzed  in  software 
error  prediction  studies) • 
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Considerations  on  the  Measure  of  Error 


to  be  Analyzed  in  Software 
Error  Prediction  Studies 


In  addition  to  the  very  complex  problem  of  the 
definition  of  an  error,  there  is  the  puzzling  problem  of 
what  measure  should  be  used  to  represent  the  errors  in  a 
routine,  module,  or  program,  once  the  data  is  collected. 

(Of  course,  this  decision  should  be  made  prior  to  data 
collection) . Is  it  the  number  of  errors  in  a program? 

If  so,  this  leaves  something  to  be  desired,  since  a very 
short  program  of  say,  ten  instructions  with  three  errors, 
would  "look"  the  same  as  the  dependent  variable  of  a very- 
large  program  of  say,  3000  instructions  with  three  errors. 

These  extremes  in  program  length  do  exist  in  the  data 
analyzed  for  sample  S and  sample  T.  The  most  desirable 
solution  to  the  problem  would  be  to  collect  data  in  such 
quantities  that  every  individual  program  length  could  be 
represented  a number  of  times,  consider  each  program 
length  a sample,  and  then  analyze  the  data  accordingly. 

Since  such  a luxury  is  not  likely  to  exist,  some  compromise 
is  required. 

Error  rate;  i.e.; errors  per  100  source  instructions, 
is  quite  commonly  used  as  an  error  measure  and  is  somewhat 
more  meaningful  than  number  of  errors.  It  still  suffers, 
however,  from  the  fact  that  a program  of  ten  instructions 
with  two  errors  receives  the  same  weight  as  a program  of 
100  instructions  with  20  errors,  one  of  1000  instructions 
with  200  errors,  etc.  If  the  regression  of  errors  as  a functic 
of  number  of  source  instructions  is  in  fact  linear,  and  the 
regression  line  goes  through  the  origin,  this  procedure 
would  be  quite  proper.  By  linear  regression  it  is  not 
meant  that  the  relationship  should  be  perfect,  but  that  a 
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straight  line  gives  as  good  a representation  of  the  rela- 
tionship as  any  curve  of  higher  degree. 

If  the  regression  is  in  fact  linear  and  the  line 
does  not  pass  through  the  origin,  a better  approach  would 
be  to  eliminate  the  linear  effect  of  number  of  source  in- 
structions on  number  of  errors  by  partial  correlation  and 
then  analyze  the  residual  error.  It  is  indicated  then,  also, 
that  the  influence  of  source  instructions  should  be  elim- 
inated from  all  other  independent  variables. 

It  would  seem  that  in  view  of  all  the  problems  inherent 
in  the  identification  and  definition  of  an  appropriate  error 
measure,  a hopeless  situation  exists.  However,  if 
the  problem  is  formulated  in  terms  of  what  variables  are 
used  to  predict  errors,  it  becomes  more  clear.  First, 
recognizing  that  number  of  errors  and  error  rate  are  two 
distinct,  though  related3,  approaches  to  error  measurement, 
two  separate  dependent  variables  exist.  To  predict  number 
of  errors,  there  are  variables  such  as  gross  characteristics 
of  program  length,  mixtures  of  instructions  of  various  types 
program  complexity  metrics,  etc.  To  predict  error  rate, 
all  of  these  variables  used  to  predict  errors,  plus  other 
program  characteristics  normalized  to  program  length  (either 
by  their  rates  per  100  source  instructions  or  by  partial 
correlation)  also  exist. 

When  error  rate  is  used  as  a measure  of  program  reli- 
ablility,  the  question  arises  as  to  whether  to  normalize 
errors  by  the  number  of  total  source  instructions,  the 
number  of  executable  source  instructions,  or  even  by  the 
number  of  generated  machine  language  instructions.  This  is 
a problem  since  the  error  measure  always  being  analyzed  is 


arelated  in  that  if  the  total  number  of  errors  in  the  progra, 
equals  zero,  the  error  rate  is  necessarily  zero. 
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the  number  of  errors  found,  and  at  any  phase  of  testing, 
there  usually  exists  some  (unknown)  number  of  latent 
errors  in  the  program(s) . 

To  clarify  this  point,  consider  that  two  programs 
(designated  as  A and  B)  exist,  each  with  400  source  in- 
structions. A had  300  executable  instructions  and  B has 
200  executable  instructions.  During  test  and  integration 
(conducted  by  automated  means)  suppose  four  errors  in  A 
and  four  errors  in  B were  found.  Normalizing  by  source 
instructions,  the  error  rate  for  A is  1.0  (error/100  source 
instructions).  For  B the  error  rate  is  also  1.0.  However, 
by  using  executable  instructions,  the  error  rate  for  A is  1.33. 
For  3 the  error  rate  is  2.0.  (Not  considered  here  is  the 
problem  of  how  many  of  the  executable  instructions  were 
actually  executed  in  the  test).  Thus,  hypothetically 
nothing  is  known  of  the  number  of  errors  in  the  non-executable 
(i.e.,  not  non-executed)  portion  of  the  programs. 

The  solution  here  is  that  the  normalizing  factor  should 
depend  upon  the  method  of  testing.  If,  for  example,  all 
the  code  is  examined  (as  in  a code  review) , the  number  of 
source  instructions  should  be  used.  If  only  the  executable 
portion  of  the  code  is  examined  for  errors,  the  number  of 
executable  instructions  should  be  used. 


APPENDIX  B 


(This  Appendix  contains  description 
of  the  predictor  variables  for 
both  samples  S and  sample  T as 
discussed  in  Section  2.0). 
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TABLE  B - 1 . SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 


VARIABLE 


DESCRIPTION 


UNNORMALIZED : 
X 1 
X 2 
X 3 
X 4 

X 5 
X 6 
X 7 

X 3 

X 9 
Xlxf 

Xll 

X 1 2 

X 1 3 


NUMBER  OF  SOURCE  INSTRUCTIONS 

NUMBER  OF  ENTRY  POINTS 

NUMBER  OF  EXIT  POINTS 

NUMBER  OF  USING  INSTRUCTIONS  WHICH 
ESTABLISH  DATA  STRUCTURE  INTERFACE 

NUMBER  OF  COMMENT  STATEMENTS 

NUMBER  OF  LABELED  SOURCE  INSTRUCTIONS 

NUMBER  OF  INSTRUCTIONS  PERFORMING  THE 
ARITHMETIC  FUNCTIONS  ADD,  SUBTRACT, 
MULTIPLY,  DIVIDE  AND  EXPONENTIATION 

NUMBER  OF  UNCONDITIONAL  BRANCH 
INSTRUCTIONS 

NUMBER  OF  CALL/LI  NR  INSTRUCTIONS 

NUMBER  OF  SYSTEM  MACROS 

..COOES  NOT  INCLUDE  THE  INSTRUCTIONS 

WHICH  ARE  GENERATED  BY  THE  MACROS) 

NUMBER  OF  USER  WRITTEN  MACROS 
(DOES  NOT  INCLUDE  THE  INSTRUCTIONS 
WHICH  ARE  GENERATED  BY  THE  MACROS) 

NUMBER  OF  EQUATE  INSTRUCTIONS  USED  TO 
EQUATE  SYMBOLS  TO  REGISTERS,  IMMEDIATE 
DATA,  OR  OTHER  VALUES 

NUMBER  OF  COMMENTED  SOURCE  INSTRUCTIONS 
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TABLE  B- 1 . SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 


VARIABLE 


DESCRIPTION 


NUMBER  OF  AND/OR  LOGICAL  CONNECTORS 

NUMBER  OF  CONDITIONAL  BRANCH 
INSTRUCTIONS 

NUMBER  OF  INVOKED  FUNCTIONS  SUCH  AS 
FLOOR , SQRT,  LOG,  ATAN  ETC. 

NUMBER  OF  INSTRUCTIONS  PERFORMING 
S CALE / ROUND  OPERATIONS 

NUMBER  OF  SHORT  DO  INSTRUCTIONS 
(WHEN  MACHINE  CODE  GENERATED  BY 
CENTRAN  DO  INSTRUCTION  WILL  BE  LESS 
THAN  17  BYTES.  USED  TO  MINIMIZE  LOOP 
EXECUTION  TIME) 

NUMBER  OF  NESTED  SHORT  DO  LOOPS 
NUMBER  OF  LOCK  MACROS 

NUMBER  OF  SOURCE  INSTRUCTIONS  WITHIN 
SHORT  DO  LOOPS 

NUMBER  OF  ADDRESS  VARIABLES  REFERENCED 

NUMBER  OF  TIMES  ADDRESS  VARIABLES  ARE 
REFERENCED 

NUMBER  OF  TIMES  ALL  BINARY  VARIABLES 
ARE  REFERENCED 

NUMBER  OF  CHARACTER  VARIABLES 
REFERENCED 

NUMBER  OF  TIMES  CHARACTER  VARIABLES 
ARE  REFERENCED 


c 

. ! ' 


TABLE  B- 1 . SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 


VARIABLE 


DESCRIPTION 


X2  7 NUMBER  OF  FIXED  POINT  VARIABLES 

.REFERENCED 

X28  NUMBER  OF  TIMES  FIXED-POINT  VARIABLES 

ARE  REFERENCED 

X29  NUMBER  OF  FLOATING-POINT  VARIABLES 

REFERENCED 

X3J  NUMBER  OF  TIMES  FLOATING-POINT 

VARIABLES  ARE  REFERENCED 

X31  NUMBER  OF  HEXADECIMAL  VARIABLES 

REFERENCED 

X32  NUMBER  OF  TIMES  HEXADECIMAL  VARIABLES 

ARE  REFERENCED 

X33  NUMBER  OF  LABELED-ARRAY  VARIABLES 

REFERENCED 

X34  NUMBER  OF  TIMES  LABE  LED- ARRAY 

VARIABLES  ARE  REFERENCED 

X35  NUMBER  OF  REGISTER  VARIABLES 

REFERENCED 

X36  NUM3ER  OF  TIMES  REGISTER  VARIABLES 

ARE  REFERENCED 

X37  NUMBER  OF  VARIABLES  WHICH  WERE 

REFERENCED  BUT  NOT  DEFINED  WITHIN  THE 
PROGRAM  (UNDEFINED  VARIABLES) 

X33  NUMBER  OF  TIMES  UNDEFINED  VARIABLES 

ARE  REFERENCED 
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TABLE  B-l.  SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 


VARIABLE 


DESCRIPTION 


X39 

TOTAL  NUMBER  OF  VARIABLES  REFERENCED 

X40 

NUMBER  OF  TIMES  ALL  VARIABLES  ARE 
REFERENCED 

X41 

NUMBER  OF  DO  LOOPS 

X 42 

NUMBER  OF  NON-NESTED  DO  LOOPS 

X43 

NUMBER  OF  DO  LOOPS  NESTED  AT  SECOND 
LEVEL 

X44 

NUMBER  OF  DO  LOOPS  NESTED  AT  THIRD 
LEVEL 

X4  5 

NUMBER  OF  DO  LOOPS  NESTED  AT  FOURTH 
LEVEL 

X46 

NUMBER  OF  DO  LOOPS  NESTED  AT  FIFTH 
LEVEL 

X47 

NUMBER  OF  DO  LOOPS  NESTED  AT  SIXTH 
LEVEL  OR  LOWER 

X43 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN  ALL 
NON-NESTED  DO  LOOPS 

X4g 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN  ALL 
SECOND  LEVEL  DO  LOOPS 

X 50 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN  ALL 
THIRD  LEVEL  DO  LOOPS 

X51 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN  ALL 
FOURTH  LEVEL  DO  LOOPS 
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TABLE  B-l. 

SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 

VARIABLE 

DESCRIPTION 

X52 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN  ALL 
FIFTH  LEVEL  DO  LOOPS 

X53 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN  ALL 
SIXTH  LEVEL  OR  LOWER  DO  LOOPS 

X54 

AVERAGE  NUMBER  OF  OPERATORS  PER 
ARITHMETIC  INSTRUCTION  (X7)  X NUMBER 
OF  SOURCE  INSTRUCTIONS 

ERRORS /PROGRAM  NUMBER  OF  ERRORS  FOUND  IN  PROGRAM 

DURING  THE  TEST  AMD  INTEGRATION  PHASE 
OF  SOFTWARE  SYSTEM  DEVELOPMENT  WHICH 
REQUIRED  A CHANGE  TO  THE  PROGRAM'S 
CODE  . 

NORMALIZED3: 

X56 

NUMBER  OF  ENTRY  POINTS /XI 

X57 

NUMBER  OF  EXIT  POINTS/X1 

X58 

NUMBER  OF  USING  INSTRUCTION  WHICH 
ESTABLISH  DATA  STRUCTURE  INTERFACE/X1 

X59 

NUMBER  OF  COMMENT  STATEMENTS /X I 

X6;T 

NUMBER  OF  LABELED  SOURCE  I NS TRUCT I ON5 /XI 

X61 

NUMBER  OF  INSTRUCTIONS  PERFORMING  THE 
ARITHMETIC  FUNCTIONS  ADD,  SU3STRACT, 
MULTIPLY,  DIVIDE  AND  EXPONENT  I AT  I ON/ X 1 

X5  2 

NUMBER  OF  UNCONDITIONAL  BRANCH 
INSTRUCTI ONS/X1 
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SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 


VAR  I A8LE 


DESCRIPTION 


X63 

NUMBER  OF  CALL/LINK  I NSTRUCT IONS/X 1 

X64 

NUMBER  OF  SYSTEM  MACROS/X1 

X6  5 

NUMEER  OF  USER  WRITTEN  MACROS/X1 

X66 

NUMBER  OF  EQUATE  INSTRUCTIONS  USED  TO 
EQUATE  SYMBOLS  TO  REGISTERS,  IMMEDIATE 
DATA,  OR  OTHER  VALUES/X1 

X67 

NUMBER  OF  COMMENTED  SOURCE  INSTRUC- 
TIONS/X1 

X6  8 

NUMBER  OF  AND/OR  LOGICAL  CONNECTORS /X 1 

X69 

NUMBER  CF  CONDITIONAL  BRANCH 
INSTRUCTIONS/ XI 

X70 

NUMBER  OF  INVOKED  FUNCTIONS  SUCH  AS 
FLOOR,  SQRT,  LOG,  ATAN  ETC. /XI 

X71 

NUMBER  OF  INSTRUCTIONS  PERFORMING 
SCALE/ROUND  OP E RAT  I CMS / X 1 

X72 

NUMBER  OF  SHORT  DO  I NS  TRUCT  I ON'S  / X 1 

X73 

NUMBER  OF  NESTED  SHORT  DO  LOOPS/X1 

X74 

NUMBER  OF  LOCK  MACROS/X1 

X75 

NUMBER  OF  SOURCE  INSTRUCTIONS  WITHIN 
SHORT  DO  LOOPS /Xl 

X76 

NUMBFR  OF  ADDRESS  VARIABLES 
REFERENCED/X1 

X77 


NUMBER  OF  TIMES  ADDRESS  VARIABLES 
ARE  REFERENCED/X1 
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SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 


VARIABLE 


DESCRIPTION 


X7S 

NUMBER  OF 
VARIABLES 

TIMES  ALL  BINARY 
ARE  REFERENCED/X1 

X79 

NUMBER  OF  CHARACTER  VARIABLES 
REFERENCED/X 1 

XS?J 

NUMBER  OF 
VARIABLES 

TIMES  CHARACTER 
ARE  REFERENCED/X1 

X81 

NUMBER  OF  FIXED-POINT  VARIABLES 
REFERENCED/ X 1 

X82 

NUMBER  OF 
VARIABLES 

TIMES  FIXED-POINT 
ARE  REFERENCED/X1 

X83 

NUMBER  OF 
VARIABLES 

FLOATING-POINT 
REFER ENCED/X1 

X84 

NUMBER  OF 
VARIABLES 

TIMES  FLOATING-POINT 
ARE  REFERENCED/X1 

X85 

NUMBER  OF  HEXADECIMAL  VARIABLES 
REFERENCED/ XI 

X36 

NUMBER  OF  TIMES  HEXADECIMAL 
VARIABLES  ARE  REFERENCED/X 1 

X87 

NUMBER  OF  LABE LE D- ARRAY 
VARIABLES  REFERENCED/X I 

X88 

NUMBER  OF  TIMES  LABE  LED- ARRAY 
VARIABLES  ARE  REFERENCED/ XI 

X89 

NUMBER  Op  REGISTER  VARIABLES 
REFEPENCED/X 1 

X9.9 

NUMBER  OF  TIMES  REGISTER  VARIABLES 
ARE  REFERENCED/X1 
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TABLE  B-l.  SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 


VARIABLE 


DESCRIPTION 


NUMBER  OF  VARIABLES  WH I CH  WERE 
REFERENCED  BUT  NOT  DEFINED  WITHIN 
THE  PROGRAM  (UNDEFINED  VARIABLE)/. 
XI 

NUMBER  OF  TIMES  UNDEFINED 
VARIABLES  ARE  REFER ENCED/X1 

TOTAL  NUMBER  OF  VARIABLES 
REFERENCED/X 1 

NUMBER  OF  TIMES  ALL  VARIABLES 
ARE  REFERENCED / X 1 

NUM3ER  OF  DO  LOOPS /XI 

NUMBER  OF  NON-NESTED  DO  LOOPS/X1 

NUMBER  OF  DO  LOOPS  NESTED  AT 
SECOND  LEVEL/X1 

NUMBER  OF  DO  LOOPS  NESTED  AT 
THIRD  LEVEL/X1 

NUMBER  OF  DO  LOOPS  NESTED  AT 
FOURTH  LEVEL/X1 

NUMBER  OF  DO  LOOPS  NESTED  AT 
FIFTH  LEVEL/X 1 

NUMBER  OF  DO  LOOPS  NESTED  AT 
SIXTH  LEVEL  OR  LOWER/X1 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN 
ALL  NON-NESTED  DO  LOOPS/X1 

NUMBER  OF  SOURCE  INSTRUCTIONS  IN 
ALL  SECOND  LEVEL  DO  LOOPS/X1 
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TABLE  B - 1 . 


SAMPLE  S PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 


VARIABLE 

DESCRIPTION 

X 1 0 4 

NUMBER  OF  SOURCE  INSTRUCTIONS 
IN  ALL  THIRD  LEVEL  DC  LOOPS /XI 

X10  5 

NUMBER  OF  SOURCE  INSTRUCTIONS 
IN  ALL  FOURTH  LEVEL  DO  LOOPS/X1 

X106 

NUMBER  OF  SOURCE  INSTRUCTIONS 
IN  ALL  FIFTH  LEVEL  DO  L0CPS/X1 

X 1 0 7 

NUMBER  OF  SOURCE  INSTRUCTIONS 
IN  ALL  SIXTH  LEVEL  OR  LOWER 
DO  LOOP S / X 1 

X 1 0 8 

AVERAGE  NUMBER  OF  OPERATORS  PER 
ARITHMETIC  INSTRUCTION  (X7)  X 
NUMBER  OF  SOURCE  I NS TRUCT I ONS / X 1 

ERROR  RATE  /PROGRAM 

NUMBER  OF  ERRORS  PER  100  LINES  OF 
SOURCE  CODE  FOUND  DURING  THE  TEST 
AND  INTEGRATION  PHASE  OF  SOFTWARE 
SYSTEM  DEVELOPMENT  WHICH  REQUIRED 
A CHANGE  TO  THE  PROGRAM'S  CODE/X1 

aALL  NORMALIZED  VARIABLE  VALUES  WERE  ACTUALLY  COMPUTED  BY 
MULTIPLYING  THE  RESPECTIVE  UNNORMALIZED  VALUES  3Y  100/X1. 
EACH  NORMALIZED  VARIABLE  IS  INTERPRETED  THEN  AS  THE  VALUE 
OF  THE  ORIGINAL  OR  UNNORMALIZED  VARIABLE  PER  100  LINES  OF 
SOURCE  CODE. 
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TABLE  3-2.  SAMPLE  T PREDICTOR  VARIABLE  DESCRIPTIONS 


VARIABLE 


DESCRIPTION 


UNNORMALIZED : 

1 . TS 

2.  LL 

.3.  IF 

4.  BR 


TOTAL  SOURCE  STATEMENTS  IN  THE 
PROGRAM  (TS=NEX+EX) 

COMPUTED  LOOP  COMPLEXITY3 

COMPUTED  IF  COMPLEXITY*3 

TOTAL  PROGRAM  BRANCHES 


5.  LS 


NUMBER  OF  LOGICAL  STATEMENTS  IN 
PROGRAM 


6 . AP 


7.  SYS 


8.  I/O 


NUMBER  OF  DIRECT  PROGRAM  INTERFACES 
WITH  OTHER  APPLICATION  PROGRAMS 
(NOT  A COUNT  OF  CALLS  TO  OTHER 
PROGRAMS) 

NUMBER  OF  DIRECT  PROGRAM  INTERFACES 
WITH  OPERATING  SYSTEM  OR  SYSTEM 
SUPPORT  PROGRAMS  (NOT  A COUNT  OF 
CALLS  TO  SYSTEM  PROGRAMS) 

NUMBER  OF  INPUT/OUTPUT  STATEMENTS 
IN  PROGRAM 


-•  C0MP  NUMBER  OF  COMPUTATIONAL  STATEMENTS 

IN  PROGRAM 

10.  DATA  NUMBER  OF  DATA  HANDLING  STATEMENTS 

IN  PROGRAM 

1 1 • NEX  NUMBER  OF  NON-EXECUTABLE  STATEMENTS 

IN  PROGRAM 


NUMBER  OF  EXECUTABLE  STATEMENTS  IN 
PROGRAM 

TOTAL  PROGRAM  INTERFACES  WITH  OTHER 
PROGRAMS  (T I =AP  + SYS) 
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SAMPLE  T PREDICTOR  VARIABLE  DESCR  I PT  I ONS 
(CONTINUED) 


VARIABLE 


DESCR I PTION 


;4.  COM  NUMBER  OF  COMMENT  STATEMENTS  IN 

PROGRAM  (COMMENTS  ARE  NOT 
INCLUDED  IN  THE  COUNT  OF  NON- 
EXECUTABLE STATEMENTS,  NEX) 

15.  RAT  AVERAGE  PROGRAMMER  RATING  (THIS 

VALUE  IS  AN  AVERAGE  BASED  ON  THE 


* 

RATINGS  OF  EACH  PROGRAMMER  WHO 
WORKED  ON  THE  PROGRAM) 

16. 

WKLD 

AVERAGE  WORKLOAD  OF  PROGRAMMERS 
WHO  WORKED  ON  THE  PROGRAM. 

17. 

ERRORS/PROGRAM 

NUMBER  OF  PROGRAMMING  ERRORS  FOUND 
IN  THE  PROGRAM  WHICH  REQUIRED  A 
CHANGE  TO  THE  PROGRAM’S  CODE 

18. 

RAT/WKLD 

RATIO  OF  AVERAGE  PROGRAMMER 
RATING  TO  AVERAGE  PROGRAMMER 
WORKLOAD 

NORMALIZED0 : 

19. 

LL/TS 

MEASURE  OF  LOOP  COMPLEXITY  PER 
100  LINES  OF  SOURCE  CODE 

20. 

IF/TS 

MEASURE  OF  IF  COMPLEXITY  PER 
100  LINES  OF  SOURCE  CODE 

21  . 

BR/TS 

NUMBER  OF  BRANCHES  PER  100  LINES 
OF  SOURCE  CODE 

22. 

LS/TS 

NUMBER  OF  LOGICAL  STATEMENTS  PER 
100  LINES  OF  SOURCE  CODE 

23 


AP/TS 


NUMBER  OF  APPLICATION  PROGRAM 
INTERFACES  PER  100  LINES  OF  SOURCE 
CODE 


TABLE  B - 2 . SAMPLE  I PREDICTOR  VARIABLE  DESCRIPTIONS 
(CONTINUED) 
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Si 
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Si 
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SL 

OL 

Si 

oL 

"CSC 

CL 

Si 

OL 
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DC 

Z 

LU 

DC 
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Gi 
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CG 
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CO 
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CQ 
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ERROR  RATE/PROGRAM  NUMBER  OF  PROGRAMMING  ERRORS  PER 

100  LINES  OF  SOURCE  CODE  PER  PROGRAM 


able  13-2 


SAMPLE  T PREDICTOR  VARIABLE  DESCRIPTIONS  (CONTINUED) 


M.  » number  of  loops  in  program  at  the  ith 
1 level  of  nesting 

= a weighting  factor 

Q » maximum  level  of  nesting  used  in  tho  system 
R » a shaping  value 

b IF 

Who  ro , 

:j,  = number  of  "IP's"  in  program  at  tho  ith 

1 level  of  nesting 

W.  = a weighting  factor,  tho  some  as  indicated 

1 for  loop  complexity  measure 

c All  normalized  variable  values  wore  computed  by 
multiplying  the  respective  unnomalizod  values  by 
iOO/TS . 
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APPENDIX  C 

(This  Appendix  contains  the  list  of 
predictor  variables  used  and  eliminated 
(a  priori)  when  predicting  errors  and 
error  rate  for  both  data  samples,  as 
discussed  in  Section  5.6). 
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TABLE  C-l 


SAMPLE  S,  LIST  OF  PREDICTOR  VARIABLES  USED  AND 
ELIMINATED  (A  PRIORI)  WHEN  PREDICTING  ERRORS/ 
PROGRAM  £ ERROR  RATE/PROGRAM 


PROJECTS 


SAMPLE  S VARIABLES 

M 

B 

P 

X 1 

SOURCE  INSTRUCTIONS 

it 

X 2 

ENTRY  POINTS 

X 3 

EXIT  POINTS 

X 4 

USING  INSTRUCTIONS 

X 5 

COMMENT  STATEMENTS 

X 6 

LABELED  INSTRUCTIONS 

X 7 

ARITHMETIC  INSTRUCTIONS 

X 8 

UNCONDITIONAL  JUMPS 

X 9 

CALLS/LINKS 

X10 

SYSTEM  MACROS 

Xll 

USER  MACROS 

X 1 2 

EQUATE  STATEMENTS 

XI 3 

COMMENTED  INSTRUCTIONS 

XI 4 

LOGICAL  CONNECTORS 

XI  5 

CONDITIONAL  JUMPS 

X16 

FUNCTIONS 

X17 

SCALING/ROUNDING  OPERATIONS 

X18 

SHORT  DO  LOOPS 

X19 

NESTED  SHORT  DO  LOOPS 

X20 

LOCK  MACROS 

1 

X21 

INSTRUCTIONS  IN  SHORT  DO  LOOPS 

1 

X2  2 

ADDRESS  VARIABLES 

X23 

ADDRESS  VARIABLE  FREQUENCY 

X2  4 

BINARY  VARIABLE  FREQUENCY 

1 

I 

1 

X25 

CHARACTER  VARIABLES 

1 

1 

1 

X26 

CHARACTER  VARIABLE  FREQUENCY 

1 

1 

1 

X27 

FIXED-POINT  VARIABLES 

X28 

FIXED-POINT  VARIABLE  FREQUENCY 

X29 

FLOATING-POINT  VARIABLES 

X30 

FLOATING-POINT  VARIABLE  FREQUENCY 

X31 

HEXADECIMAL  VARIABLES 

1 

1 

1 

X3  2 

HEXADECIMAL  VARIABLE  FREQUENCY 

1 

1 

1 

X33 

LABELED-ARRAY  VARIABLES 

X 3 4 

LABELED-ARRAY  VARIABLE  FREQUENCY 

2 

2 

2 

X35 

REGISTER  VARIABLES 

X36 

REGISTER  VARIABLE  FREQUENCY 

X37 

UNDEFINED  VARIABLES 

X38 

UNDEFINED  VARIABLE  FREQUENCY 

X39 

TOTAL  VARIABLES 

3 

3 

3 

X40 

TOTAL  VARIABLE  FREQUENCY 

3 

3 

3 
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TABLE  C-l. 


SAMPLE  S,  LIST  OF  PREDICTOR  VARIABLES  USED  AND 
ELIMINATED  (A  PRIORI)  WHEN  PREDICTING  ERRORS/ 
PROGRAM  £ ERROR  RATE/PROGRAM  (CONTINUED) 


§ 

i< 


| 

v 

I 

I 

1 

I 

i 

t 


f 

ti 


t 

s 


i 

f 

t 


I 


PROJECTS 


SAMPLE  S VARIABLES 

M 

B 

p 

X41 

TOTAL  DO  LOOPS 

3 

3 

3 

X42 

NON-NESTED  DO  LOOPS 

X43 

DO  LOOPS  NESTED  AT  2ND  LEVEL 

X44 

DO  LOOPS  NESTED  AT  3RD  LEVEL 

X45 

DO  LOOPS  NESTED  AT  4TH  LEVEL 

X46 

DO  LOOPS  NESTED  AT  5 TH  LEVEL 

X47 

DO  LOOPS  NESTED  AT  6 TH  LEVEL  OR  LOWER 

X48 

INSTRUCTIONS  IN  NON  NESTED  DO  LOOPS 

X49 

INSTRUCTIONS  IN  2ND  LEVEL  DO  LOOPS 

X50 

INSTRUCTIONS  IN  3RD  LEVEL  DO  LOOPS 

X5I 

INSTRUCTIONS  IN  4TH  LEVEL  DO  LOOPS 

X 5 2 

INSTRUCTIONS  IN  5 TH  LEVEL  DO  LOOPS 

X53 

INSTRUCTIONS  IN  6TH  LEVEL  OR  LOWER 

X54 

DO  LOOPS 

SOURCE  INSTRUCTIONS  X AVERAGE  NUMBER 

X55 

OF  OPERATORS/ARITHMETIC  INSTRUCTION 
NO.  OF  PROGRAMMING  ERRORS  FOUND  DURING 
THE  TEST  5 INTEGRATION  (TSI)  PHASE 

DEPENDENT  VARIABLE 
ERRORS/PROGRAM 

X56 

ENTRY  POINTS/X1 

X57 

EXIT  POINTS/X1 

X58 

USING  INSTRUCTIONS/X1 

X 59 

COMMENT  STATEMENTS /XI 

X60 

LABELED  I NS TRUC T I ONS /X 1 

X61 

ARITHMETIC  I NS TRUCT I ONS / X 1 

X6  2 

UNCONDITIONAL  JUMPS/X1 

X63 

CALLS / L I NK  S / X 1 

X64 

SYSTEM  MACROS /XI 

X6  5 

USER  MACROS/X  I 

X66 

EQUATE  STATEMENTS/X  1 

. 

X67 

COMMENTED  I NS TRUCT I ONS /X 1 

X68 

LOGICAL  CONNECTORS /XI 

X69 

CONDITIONAL  JUMPS/XI 

X 70 

FUNCT I ONS /XI 

X71 

SCALING/ROUNDING  OPERAT I ONS /X 1 

X72 

SHORT  DO  LOOPS /XI 

X73 

NESTED  SHORT  DO  LOOPS/X1 

X74 

LOCK  MACROS /XI 

i 

X75 

INSTRUCTIONS  IN  SHORT  DO  LOOPS/X1 

i 

X76 

ADDRESS  VARIABLES/X1 

X 7 7 

ADDRESS  VARIABLE  FREQUENCY/X 1 
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TABLE  C - 1 , 


SAMPLE  S,  LIST  OF  PREDICTOR  VAR  I AB 
ELIMINATED  (A  PRIORI)  WHEN  PREDICT 
PROGRAM  5 ERROR  RATE/PROGRAM  (CONT 


LES  USED  AND 
ING  ERRORS/ 

I NUED) 


PROJECTS 


SAMPLE  S VARIABLES 

M 

B 

P 

X7  8 

BINARY  VARIABLE  FREQUENCY/X1 

1 

1 

1 

X79 

CHARACTER  VARIABLES/X1 

1 

1 

1 

X80 

CHARACTER  VARIABLE  FREQUENCY /X 1 

1 

1 

1 

X81 

FIXED-POINT  VAR  I AB  LES / X 1 

X82 

FIXED-POINT  VARIABLE  FREQUENCY/X1 

X83 

FLOATING-POINT  VARIABLES/X1 

X84 

FLOATING-POINT  VARIABLE  FREQUENCY/ X 1 

X85 

HEXADECIMAL  VARIABLES/X1 

1 

1 

1 

X86 

HEXADECIMAL  VARIABLE  FREQUENCY /XI 

1 

1 

1 

X87 

LABELED-ARRAY  VARIABLES/X1 

X88 

LABELED-ARRAY  VARIABLE  FREQUENCY/X1 

2 

2 

2 

X89 

REGISTER  VAR  I AB  LES / X 1 

X90 

REGISTER  VARIABLE  FREQUENCY/X1 

X91 

UNDEFINED  VARIABLES/X1 

X92 

UNDEFINED  VARIABLE  FREQUENCY /XI 

X9  3 

TOTAL  VARI AB  LES/X1 

3 

3 

3 

X94 

TOTAL  VARIABLE  FREQUENCY /XI 

3 

3 

3 

X95 

TOTAL  DO  LOOPS /X 1 

3 

3 

3 

X9& 

NON-NESTED  DO  LOOPS/X1 

X97 

DO  LOOPS  NESTED  AT  2ND  LEVEL/X1 

X98 

DO  LOOPS  NESTED  AT  3RD  LEVEL/X1 

X99 

DO  LOOPS  NESTED  AT  4TH  LEVEL/X1 

X 100 

DO  LOOPS  NESTED  AT  5TH  LEVEL/X1 

X 1 0 1 

DO  LOOPS  NESTED  AT  6 TH  LEVEL  OR 

X10  2 

LOWER/X1 

INSTRUCTIONS  IN  NON  NESTED  DO 

X10  3 

LOOPS /XI 

INSTRUCTIONS  IN  2ND  LEVEL  DO  LOOPS/X1 

X 104 

INSTRUCTIONS  IN  3RD  LEVEL  DO  LOOPS/X1 

X105 

INSTRUCTIONS  IN  4TH  LEVEL  DO  LOOPS/X1 

X106 

INSTRUCTIONS  IN  5 TH  LEVEL  DO  LOOPS/X1 

__  . _ __ 

X 10  7 

INSTRUCTIONS  IN  6 TH  LEVEL  OR  LOWER 

X 1 0 8 

DO  LOOPS /XI 

SOURCE  INSTRUCTIONS  X AVERAGE  NUMBER 

X109 

OPERATORS /AR I TH.  1 NSTRUCT I ON/X 1 
(NO.  OF  PROGRAMMING  ERRORS  FOUND 
DURING  TSI  PHASE ) /X 1 

DEPENDENT  VARIABLE 
ERROR  RATE/PGM. 
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TABLE  C-l.  SAMPLE  S,  LIST  OF  PREDICTOR  VARIABLES  USED  AMD 
ELIMINATED  (A  PRIORI)  WHEN  PREDICTING  ERRORS/ 
PROGRAM  £ ERROR  RATE/PROGRAM 


FOOTNOTES 


!s  THE  EMPTY  CELLS  IN  THIS  TABLE  ARE  USED  TO  IDENTIFY  THOSE 
PREDICTOR  VARIABLES  THAT  WERE  MADE  AVAILABLE  FOR  AUTO- 
MATIC SELECTION  BY  THE  STEPWISE  REGRESSION  PROCEDURE  FOR 
ENTRY  INTO  THE  PREDICTION  EQUATION. 

A PRIORI  VARIABLE  ELIMINATION  CRITERIA 


1  THE  VARIABLES'  DATA  VALUES  WERE  ALL  ZERO  IN  THE  SAMPLE. 
(APPARENTLY,  THE  VARIABLES  COULD  HAVE  BEEN  UNAVAILABLE, 
NOT  COLLECTED  OR  COUNTED  DURING  THE  PROGRAM  SCANNING 
OPERATION,  NON-EXISTENT,  OR  NOT  APPLICABLE  IN  THE 
PROGRAMS  THAT  WERE  USED  FOR  THIS  ANALYSIS). 


2  THE  VARIABLE  WAS  HIGHLY  CORRELATED  WITH  ANOTHER  PREDICTOR 
VARIABLE.  (NOTE:  THE  CORRELATION  COEFFICIENTS,  R 

AND  RX87/XB8>  WERE  FOUND  TO  BE  1.00  IN  EACH  PROJECT5’ ' *3  H 

SAMPLE;  THEREFORE  ONLY  ONE  VARIABLE  FROM  EACH  PAIR, 

X33  AND  X8  7 RESPECTIVELY,  WERE  MADE  AVAILABLE  FOR 
SELECTION  IN  THE  STEPWISE  REGRESSION  PROCEDURE). 


3  THE  VARIABLE  IS  A LINEAR  COMBINATION  OF  OTHER  PREDICTOR 
VARIABLES.  NOTE: 


X39 
X40 
X41 
X93 
X9  4 
X95 


X2  2 + X 2 5 + X 2 7 + 
X2  3 + X 2 4 + X 26  + 
X 1 8 + X 1 9 + X42  + 
X76  + X 79  + X8  1 + 
X7  7 + X 7 8 + X80  + 
X72  + X73  + X96  + 


X 2 9 + X 3 1 + X 3 3 
X 2 8 + X 3 0 + X 3 2 
X 4 3 + X 4 4 + X 45 
X 8 3 + X8  5 + X87 
X 3 2 + X 84  + X 86 
X97  + X98  + X99 


+ X 3 5 + X 3 7 

+ X34  + X36  + X38 

+ X 46  + X47 

+ X89  + X91 

+ X 8 8 + X90  + X92 

+ XI 00  + XI 01 
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TABLE  C-2 . SAMPLE  T,  LIST  OF  PREDICTOR  VARIABLES  USED  AND 
ELIMINATED  (A  PRIORI)  WHEN  PREDICTING  ERRORS/ 
PROGRAM 


PROGRAM 

STRUCTURE 

+ 

PROGRAMMER 

VARIABLES 

PROGRAM 

STRUCTURE  VARIABLES 

ONLY 

SUBSYSTEM 

SUBSYSTEM 

VARIABLE 

A 

B 

C 

D 

E 

F 

A 

B 

C 

D 

E 

F 

G 

H 

1 . 

TS 

:: 

2 . 

LL 

3 . 

I F 

4. 

BR 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

5 . 

LS 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

6 . 

AP 

7 . 

SYS 

3. 

I /O 

9 . 

COMP 

10. 

DATA 

1 

1 

1 

1 

1 

1 

11 . 

NEX 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

12  . 

EX 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

13  . 

T 1 

2 

2 

2 

2 

O 

A. 

2 

2 

2 

2 

2 

2 

2 

2 

2 

14. 

COM 

13  . 

RAT 

3 

3 

3 

3 

3 

3 

3 

3 

16. 

WKLD 

3 

3 

3 

3 

3 

3 

3 

3 

17  . 

RAT/ 

3 

3 

3 

3 

3 

3 

3 

3 

WKLD 

13  . 

LL/TS 

19  . 

IF /TS 

21!  . 

BR/TS 

21  . 

LS/TS 

2 2 . 

AP/TS 

23  . 

SYS/TS 

2 4 . 

IO/TS 

23  . 

COMP/TS 

2 6 . 

DATA/TS 

27  . 

NEX/TS 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 3 . 

EX/TS 

29  . 

T I /TS 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

30  . 

COM/TS 

TOTAL 

23  23 

23 

23 

23 

2 3 

20 

2 0 

20 

2 0 

20 

2 0 

20 

2 0 

PRED I C- 

TORS  USED 

— 
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TABLE  C-2.  SAMPLE  T,  LIST  OF  PREDICTOR  VARIABLES  USED  AND 
ELIMINATED  (A  PRIORI)  WHEN  PREDICTING  ERRORS/ 
PROGRAM  (CONTINUED) 


"THE  EMPTY  CELLS  IN  THIS  TABLE  ARE  USED  TO  IDENTIFY  THOSE 
PREDICTOR  VARIABLES  THAT  WERE  MADE  AVAILABLE  FOR  AUTOMATIC 
SELECTION  BY  THE  STEPWISE  REGRESSION  PROCEDURE  FOR  ENTRY 
INTO  THE  PREDICTOR  EQUATION. 


A PRIORI  VARIABLE  ELIMINATION  CRITERIA 

WhE  VARIABLE  WAS  HIGHLY  CORRELATED  WITH  ANOTHER  PREDICTOR 
VARIABLE . 

2THE  VARIABLE  IS  A LINEAR  COMBINATION  OF  OTHER  PREDICTOR 
VARIABLES . 

NEX  = TS-EX 
T I = AP+SYS 

NEX/TS  = 1-EX/TS 
TI/TS  = AP/TS+SYS/TS 

3THE  VARIABLE  WAS  NOT  APPLICABLE  FOR  THIS  SET  OF  COMPUTER  RUNS. 

(NOTE  - ONE  OF  THE  FOUR  PREDICTORS  (BR,  LS,  DATA,  EX)  HAVING 
THE  HIGHEST  CORRELATION  WITH  ERRORS,  AND  THE  TS  VARI- 
ABLE, WERE  BOTH  MADE  AVAILABLE  FOR  SELECTION.  EACH 
PREDICTOR  THEN  COULD  BE  SWAPPED  FOR  THE  OTHER  (SINCE 
ALL  FIVE  PREDICTORS  WERE  HIGHLY  CORRELATED)  USING  THE 
F SWAP  SELECTION  ALGORITHM). 
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TABLE  C-3 


SAMPLE  T_,  LIST  OF  PREDICTOR  VARIADLES  USED  AND 
ELIMINATED  CA  PRIORI)  WHEN  PREDICTING  ERROR 
RATE/PROGRAM 


PROGRAM 

STRUCTURE 

4- 

PROGRAMMER 

VARIABLES 

PROGRAM 

STRUCTURE 

ONLY 

VARIABLES 

SUBSYSTEM 

SUBSYSTEM 

VARIABLE 

A 

B 

C 

D 

E 

P 

A 

B 

C 

D 

E 

F 

G 

H 

I . 

TS 

3( 

2 . 

LL 

3 . 

IF 

4. 

BR 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

5 . 

LS 

1 

1 

1 

1 

1 

1 

G, 

AP 

7. 

SYS 

8. 

I/O 

9 . 

COMP 

10. 

DATA 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 1 . 

NEX 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

12. 

EX 

1 

1 

l 

1 

i 

1 

1 

1 

1 

1 

1 

1 

13. 

T I 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

14. 

COM 

13  . 

RAT 

3 

3 

3 

7 

3 

3 

3 

3 

16  . 

WKLD 

3 

3 

3 

3 

3 

3 

3 

3 

17  . 

RAT/ 

3 

3 

3 

3 

3 

3 

3 

3 

WKLD 

* 

IS  . 

LL/TS 

1 9 . 

IF/TS 

20. 

BR/TS 

21  . 

LS  /TS 

22  . 

AP/T3 

23  . 

SYS/TS 

24. 

IO/TS 

25. 

COMP/TS 

26. 

DATA/TS 

27  . 

NEX/TS 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

23  . 

EX/TS 

2 9 . 

T I /TS 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

3 0 . 

COM/TS 

TOTAL 

23 

23 

23 

23 

23 

23 

20 

20 

20  20 

20 

20 

20 

20 

PREDI C- 

TORS  USED 

TABLE  C-3.  SAMPLE  T,  LIST  OF  PREDICTOR  VARIABLES  USED  AND 
ELIMINATED  (A  PR  I OR  I 5 WHEN  PREDICTING  ERROR 
RATE/PROGRAM  CCONTINUED) 


"THE  EMPTY 
PREDICTOR 
SELECTION 
INTO  THE 


CELLS  IN  THIS  TABLE  ARE  USED  TO  IDENTIFY  THOSE 
VARIABLES  THAT  WERE  MADE  AVAILABLE  FOR  AUTOMATIC 
BY  THE  STEPWISE  REGRESSION  PROCEDURE  FOR  ENTRY 
PREDICTION  EQUATION. 


A PRIORI  VARIABLE  ELIMINATION  CRITERIA 


^HE  VARIABLE  WAS  HIGHLY  CORRELATED  WITH  ANOTHER  PREDICTOR 
VARIABLE . 


2THE  VARIABLE 
VARIABLES  . 


IS  A LINEAR  COMBINATION  OF  OTHER  PREDICTOR 


NEX 
T I 

NEX/TS 

TI/TS 


TS-EX 

AP+SYS 

1-EX/TS 

AP/TS+SYS/TS 


3T'H£  VARIABLE  WAS 


NOT  APPLICABLE  FOR  THIS  SET  OF  COMPUTER  RUNS. 


(NOTE  - ONE  OF  THE  FOUR  PREDICTORS  (BR,  LS,  DATA,  EX)  HAVING 
THE  HIGHEST  CORRELATION  WITH  ERROR  RATE, AND  THr  TS 
VARIABLE,  WERE  BOTH  MAOE  AVAILABLE  FOR  SELECTION. 
EACH  PREDICTOR  THEN  COULD  BE  SWAPPED  FOR  THE  OTHER 
(SINCE  ALL  FIVE  PREDICTORS  WERE  HIGHLY  CORRELATED) 
USING  i . : E F SWAP  ALGORITHM). 
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METRIC  SYSTEM 


BASE  UNITS: 

(Quantity 


length 

mass 

time 

electric  current 
thermodynamic  temperature 
amount  of  substance 
luminous  intensity 

SUPPLEMENTARY  UNITS; 

plane  angle 
solid  angle 

DERIVES  UNITS: 

Acceleration 

activity  (of  a radioactive  sourcel 

angular  acceleration 

angular  velocity 

urea 

density 

electric  capacitance 

electrical  conductance 

electric  field  strength 

electric  inductance 

electric  potential  difference 

electric,  resistance 

electromotive  force 

energy 

entropy 

force 

frequency 

illuminance 

luminance 

luminous  flux 

magnetic  field  strength 

magnetic;  flux 

magnetic  (lux  density 

magnetomotive  force 

power 

pressure 

quantity  of  electricity 
quantity  of  heat 
radiant  intensity 
specific  heat 
stress 

thermal  conductivity 
velocity 

viscosity,  dynamic 

viscosity,  kinematic 

voltage 

vole  me 

wuvenumlxrr 

work 


Unit 


motre 

kilogram 

second 

ampere 

keivin 

mole 

candela 


radian 

steradian 


metre  per  second  squared 

disintegration  per  second 

radian  per  second  squared 

radian  per  second 

square  metre 

kilogram  per  c.ubit;  metre 

farad 

siemens 

volt  per  metre 

henry 

volt 

ohm 

volt 

joule 

joule  per  keivin 

newton 

hertz 

lux 

candela  per  square  metre 
lumen 

ampere  par  metre 

wnber 

tesla 

ampere 

watt 

pascal 

coulomb 

joule 

watt  per  steradian 
loule  per  kilogram  keivin 
pasi  al 

wall  per  melre-kelvin 
metre  per  second 
pasc  iil-second 
square  metre  per  second 
volt 

cubic-  metre 
reciprocal  metre 
joule 


SI  PREFIX ES; 


Multiplication  Fac  tors 

i poo  txin  mm  non  to1' 

1 OOP  OOP  OOP  HI'' 

1 OOP  not)  IP'' 

1 111)0  - IP' 
.00  IP' 

IP  IP' 

0 1 IP  1 

PPt  IP  1 
0 0(11  IP  ' 

i)  pup  poi  ip  * 

u.owionnoni  ip  “ 
o.ooo  pop  pop  tun  in  ” 

p otioooo non  poo  oot  ip  " 

tl  (IPO  POP  OOP  1)011  POP  001  10  111 

* III  he  avoided  where  possible 


SI  Symbol 

m 

kg 

s 

A 

K 

mol 

c.d 


rad 

sr 


F 

S 

>1 

V 

V 


N 

Hz, 

lx 

lm 

Wb 

T 

A 

W 

I’a 

C 

I 


I’a 


V 


Formula 


m/s 

(disintegration  )/s 

rad/5 

rad/s 

m 

kg/m 

A-s/V 

A/V 

V/m 

V-s/A 

W/A 

V/A 

W/A 

N-m 

J/K 

kg.m/s 

(cycle)ls 

im/m 

cd/m 

cd’sr 

A/m 

V-s 

Wb/tn 

|/s 

N/m 

A-s 

N-m 

W/sr 

J/kg-K 

Nwi 

W/m-K 

m/s 

Pas 

m/s 

W'A 

ni 

(wuve)/m 

Nun 


Prefix 

SI  Synilw 

I urn 

T 

C ' 

K 1 H® 

nmHQ 

M 

kilo 

k 

h*»c:lo* 

h 

ilnka* 

da 

cUn:l  * 

<1 

irntr 

c 

in  ii  1 i 

m 

micro 

M 

nnnn 

n 

pi  CO 

fomto 

P 

f 

Httl) 

a 

r ' 

»-■ 


MISSION 
of 

Rime  Air  Development  Center 


RADC  plans  and  conducts  research,  exploratory  end  advanced 
development  programs  in  command,  control,  and  communications 
(C3)  activities , and  in  the  C3  areas  or'  information  sciences 
and  intelligence.  The  principal  technical  mission  areas 
are  communications , electromagnetic  guidance  and  control, 
surveillance  of  ground  and  aerospace  objects,  intelligence 
data  collection  and  handling.  Information  system  technology, 
ionosDheric  propagation,  solid  state  sciences,  microwave 
physics  and  electronic  reliability,  maintainability  and 
compatibility. 


